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INTRODUCTION 



Conference Overview 

Recent recommendations to improve education have come from a wide set of 
perspectives including legislatures, state departments of education, boards of 
education, local education agencies and business. Often these efforts are 
implemented independently of one another and rely on widely different sources 
of supporting data. In some cases, educational agencies have minimal data 
available to judge the results of their efforts. In others, agencies are 
uncertain about what data are appropriate or how they should be used to judge 
educational improvements. 

This conference, which was supported by NWREL with funding from the 
National Institute of Education, was built on the premise that an open 
discussion of approaches to using data will enable diverse groups to work more 
efficiently and cooperatively. This in turn will support the implementation 
and assessment of school improvement and reform efforts. 



Conference Objectives Were : 

• Share the perspectives of policy makers, educators and business on 
what data should be used in planning, implementing and evaluating 
educational reforms 



Describe how different agencies can work cooperatively to efficiently 
share information in order to improve our educational systems 

Illustrate through specific examples how data systems can serve those 
concerned with school improvement. 



Presentation Abstracts 



Dean Hosier , representing the Oregon Business Council, spoke of Oregon's need 
for broad based discussion and decision making regarding Oregon 9 s future, 
particularly its economic future. He described the role of the Oregon 
Business Council, an organization made up of key business leaders in the 
State, in helping to shape visions and plans for Oregon and the Northwest 
region* Hosier spoke of the Council's need for support from the education 
community and invited educators to work with the Council to address current 
needs and problems. 



Hartha Darling, of the Washington Business Roundtable, spoke of the findings 
from the Itound table's education study within the State of Washington. She 
described the mail, telephone and in-person surveying the Roundtable conducted 
to gather information about attitudes toward Washington's education system. 
Darling also described the Round table's investigation of testing procedures 
and recoMendations the Roundtable based on findings in this and other areas. 



Donald B. Bgge , of the Oregon Department of Education, emphasized the 
importance of having accurate data and information for educational decision 
making. Bgge presented and discussed a list of indicators used by selected 
states for measuring educational effectiveness. Bgge gave particular emphasis 
to the importance of selecting indicators which accurately portray the school, 
district, state or other educational system being described. Examples of 
needed improvements in collecting and sharing data were presented. 



Michael Kirst , of Stanford University, called attention to the ways that 
changes in educational priorities influence states and districts to collect 
and display different kinds of data. He emphasized that expanded data 
collection/analysis responsibilities require additional staff and support. 
Kirst discussed seven approaches to collecting information and using it to 
measure school effectiveness, trends, comparative benefits, and so on. A 
handout displaying data collected and shared by the State of California was 
distributed and discussed. 



Francis Hatanaka , from the Hawaii Department of Education, described the role 
of data and school profiles in supporting accountabilty. Hawaii has 
established state level goals on several indicators, provides information to 
districts and schools and is striving to be more accountable for education 
within Hawaii. 



Harvey Cromett , Alaska Department of Education, described establishing a data 
base at the state level. Alaska is also developing a system for establishing 
and reporting school profiles using Apple computers to facilitate 
communication between districts and the State Department. 



Ichiro F ukugoto y of the Hawaii Department of Education, discussed the ways 
that Hawaii's educational data system is used. Pukumoto shared and discussed 
a series of school profile charts and talked about both the anticipated and 
unforeseen uses to which this information is put. Major uses of the data 
include accreditation studies, school- level goal setting and planning, and 
helping parents decide where to enroll their children. 



Wayne Neuburqer. Oregon Department of Education, addressed the issue of 
establishing goals for 1390 and detailed the preparations for school level 
profiles which will enable the schools to see the relationship between their 
data and statewide data. Be reviewed the work of task forces that made 
recommendations for the Oregon Action Plan, and emphasized the need for the 
state to support change at the school a* * classroom level. 



Stephen Slater, Oregon Department of Education, talked on Oregon's vision of 
school improvement and the state's role in supporting school improvement. He 
pointed out efforts to develop troubleshooting techniques at the local level 
through assistance in profile development and data analysis. 



Dennis Deck, Northwest Regional Educational Laboratory, spoke about the High 
School and Beyond Study and its use at the local level. He showed how local 
data compare with national data with the High School and Beyond Study, how 
this information is used to supplement achievement data, and pointed out 
problem areas such as a need for comparative data over several years. 



Don Holznagel , Northwest Regional Educational Laboratory, spoke on issues 
related to establishing and maintaining regional data bases. His presentation 
covered the purpose, organization, validity, reliability and accessibility 
characteristics of data bases. 



William Cooley , from the University of Pittsburgh, discussed the events and 
issues connected with establishing a data management system in the Pittsburgh 
schools. Bis presentation focused on ways to ensure that data are useful and 
used, how teachers and planners can use the kinds of data in the system, and 
the resistance of seme groups to the collection and sharing of these data. 



Robert Blum , Northwest Regional Educational Laboratory, explained profile 
development and goal setting for school improvement. The responsibilities of 
the Leadership Team are defined and examples are given of how this process for 
school improvement works with emphasis on the individual school. 



Michael Brott , Valley Education Consortium, presented a description of how a 
goal directed/outcome based curriculum and assessment system has been 
developed and implemented, with emphasis on a three-level monitoring system. 
He described the roles of both building administrators and teachers in 
organising and using data to identify those students to be placed on a watch 
list to receive help through special programs (i.e., Chapter 1, special 



FRIT education, handicapped, or tutorial) to supplement that student's regular 
classroom wprk. w t . , / r% 



Robert Haggond, who works with the Springfield School District, described the 
system for collecting, displaying and using data in that district. In 
particular, Barmond described the development of this system and then reviewed 
with participants a data summary chart used to arrive at decisions about the 
need for program improvement efforts. 



Milton Snyder, Federal Way School District in Washington State, presented on 
how his district developed an individual school learning plan to help 
revitalize the district's schools. He detailed the funding problems, test 
score changes, and efforts to obtain hiihly skilled personnel to implement the 
goals that have been established. 



Matthew Prophet. Portland Public Schools, spoke about the needs for evaluation 
and assessment data in a large district and the uses of this data in 
reorganizing the Portland School system. He also described the history and 
current status of an instructional technology plan. 



The remainder of the conference proceedings is an edited set of the 
presentations made by the conference participants. 
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A Business Council's Needs for Education Data 

Dean Nosier 
Oregon Business Council 



I am complimented to be your morning speaker, and before we get into my 
(' talk I woa-d like to wish you and your children a safe but scary Halloween. 

I would like to talk this morning about the Oregon Business Council. 
The Oregon Business Council is a nonprofit, nonpartisian, independent 
organisation of Oregon business executives. We were founded originally with 
20 directors, primarily out of the Portland metropolitan area. Our fiscal 
year began in April. We now have 32 directors and are expanding our regional 
representation. The directors are all chief executive officers of major 
private and publicly held corporations, when they are not at the meetings, 
these representatives cannot forward a proxy vote; we are structured this way 
to encourage maximum participation and members have agreed to have this 
stipulation in the by-laws. 

The Oregon Business Council is patterned after an emerging concept of 
state round tables, or business councils, or business partnerships — whatever 
you want to call thera. There are now about 15 such organizations nationally. 
The first one was in Minnesota in 1975; California developed its roundtable 
the next year; Pennsylvania in 1979; New Hampshire, the State of Washington, 
f * and approximately a dozen others since then. 

The mission of the Oregon Business Council is similar to other such 
councils: To focus the talents, time and resources of its directors on issues 
of broad importance to Oregon's long-term social and economic well-being. To 
present to Oregon! an s constructive ideas. To resolve each issue addressed, 
basing all recommendations on thorough research and personal involvement by 
each director. To encourage public debate and foster policy consensus among 
business and government, education and diverse interest groups throughout the 
state. 

The more established organizations have worked on projects such as health 
care cost containment, educational enhancement, public finance, pension plan 
management, economic development, worker's compensation, tourism and so on. 
The list is extensive. 



Why the Oregon Business Council? Why now? Why is this trend occur ing? 

As we all are aware, there is significant change going on in the federal 

government. There is more responsibility being turned back to the states, 
often without revenue. 

In the State of Washington, a roundtable was established about two and a 
half years ago, and it was probably another two years in forming. The same 
thing has happened here in Oregon. One of the things that is curiously 
different about Washington and Oregon, however, is that, at the Washington 
Roundtable, when George Weyerhauser and T.A. Wilson of Boeing speak, it's like 
E.F* Button commercials: everyone listens. In Oregon it is not quite that 
way. We have a lot of balanced peer power, if I can use that term, and a real 
disparity of interest in what the problems are. If you go to the high tech 
community, it's predominately education and particularly higher education. If 
,hKJC you talk about the timber industry, it\* the allowable cut. The issues go on 



and on. In Oregon about two and a half years ago, a series of articles and 
editorials in the Oregon ian and Business Journal discussed the perceived lack 
of private sector leadership, some of the editorials spoke of "empty saddles" 
or "leadership vacuums." And at the same time there were tremendous swings in 
the economy— from an Oregon peak in 1979 to a recession that has been going on 
since that time. Also significant changes in the behavior of legislatures in 
Oregon, Washington and other states. 

In Oregon there continues to be a perceived lack of direction. I have 
interviewed many people now from all walks and diciplines, and the one thing 
that I recognize is this constant frustration when trying to identify a 
leader. The Glenn Jacksons and the Tom McCalls are gone, and the void hasn't 
been filled with some magnetic personality. There seems to be no 
coordination, no blueprint; communication is ragged, and there is no vision. 
Oregon has a more unstable economy than Washington, and we do not have a 
Boeing, so we do not have our timber industry problems offset by a major air 
frame industry. Even Idaho is doing better in many catagories than we are. 
The state's five-year forecast that we presented to the directors a week ago 
Wednesday indicated a bleak future for this state. Such a forecast is 
obviously driven by attempts to estimate revenue, but we could easily see that 
the next five years are going to be below the national average. Tn fact, it 
will take five more years to get to the '79 peak and we will still be eight 
percentage points behind the U.S. national average in growth. The«3 projects 
are of great concern to the directors. 

There will be continued losses of jobs in our resource and commodity-based 
industries like timber, agriculture and mining, and those will be permanent 
losses requiring significant retraining. High tech will not replace those 
losses and particulary not at the same valu£. So we are going to experience 
massive restructuring in the State of Oregon and move very rapidly from the 
industrial to service to the information economy. 

The futurists tell u? that education will be the largest business in the 
U.S., and I believe that, particularly when I look at the forecasts. The 
legislature in Oregon set up an Oregon Futures Commission two years ago and it 
is now concluding its work and will be submitting a report to the legislature 
in January 1986. Their forecast indicates a dramatic series of changes for 
many industries and services in the next 25 years. Education, as you know, 
will be going through profound changes. The demand for educational services 
will be enormous. All solutions to problems that are projected indicate a 
need for education— for the unemployed logger, the millworker, the warmer, the 
manufacturing line worker, the high tech engineer with a half-life of about 
three years, and the many people who will be stymied in their career growth 
and will return to school for retraining or education in different career 
pursuits. Entrepreneur ship will be a driving force for our economy, moving 
from the service to the information economy. 

We recognize that the education community is a source of knowledge that 
the Oregon Business Council desperately wants as a partner. We are looking to 
you to assist us in research and debate, and to create forms, and to join us 
in communicating our needs to the rest of the state population and government. 
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We anticipate putting together a long-range plan in order to stimulate a 
vision and direction for Oregon. So we are taking a look at all the primary 
indicator areas and trying to identify one to three of the most significant 
actions within that indicator, and then trying to prioritize those into a very 
condensed list. At that point, we are coming back to you, the education 
community, and we are not going to be bashful about asking for support, help 
and research effort to aid us in understanding those key areas better. We 
want to take that agenda out to government and other areas and ask for 
support, understanding and debate in order to create some compromises that 
will be acceptable to us all. 

The other thing that I found is that Oregon didn't have the magnet issue 
that Washington had. The Washington Roundtable had an immediate issue with 
bond ratings — WPPSS, the funding of education, the e-x>noray downturn, the sales 
tax wasn't supplying adequate program revenues. Therefore, they immediately 
gravitated around that and the impact on education. So they had two pressing 
target issues initially that really got them formed. In Oregon we've been 
struggling with a tremendous array of issues. But before we start looking for 
these long-term solutions and strike up our alliances, I would like to ask you 
some questions, because we have been asking them of ourselves and they reveal 
the difficulty of trying to get an organization like this primed and moving. 
Collectively, what do we really want for our state? Do we want economic 
development? Do we want quality education for our children? Do we want to 
pay for the necessary and consistent public services and infrastructure that 
goes with it? Are we willing to accept less than the national average in 
economic growth and prosperity? How do we come to a workable compromise on 
key environmental, social, political and economic development issues? If you 
don't have a magnet leader, where and when do all important constituent groups 
in Oregon sit down together at the same table to hammer out an action agenda 
for Oregon's economic future? When will we then sit down as a region of 
Oregon, Washington and Idaho and deal with our similar problems? How do we 
develop consensus and communicate a shared vision for cur future? Where does 
the leadership come from to make this vision a reality? Government? Business? 
Education? Other groups? Can we forge a new partnership for Oregon's economic 
future? 

Well, at the Oregon Business Council we believe there can be a high level 
of cooperation between the public and private sectors of this state, although 
we don't claim to have the answers at this point. But we are working to 
understand the economy better and to define its priorities, and we do realize 
that there is a need for vision and an agenda of critical issues, and there is 
a need for goals and an action plan. And particulary, there is a need for 
communication among business, government, educators, labor and the key 
stakeholers to create an understanding and a focus. There is a dramatic need 
for a solid partnership to attack the issues that we face in this state. The 
OBC intends to create that agenda and bring it to a common table, not with the 
idea that w* have the finalized agenda, but rather to have something we can 
sit down and deal with and try to come up with compromises. The vision 
involves economic prosperity for Oregon, while maintaining a high quality of 
life for our citizens, but this won't happen on its own. The issues we are 
grappling with today, such as balancing the tax structure, funding quality 
education and revitalizing our ailing, basic industries, are all part of the 
bigger picture. 



10 



The Oregon Business Council will work with the rest of the state, and 
particularly education, to develop a collective vision for Oregon's future, 
help to frame the targets and write an implementation plan called an "Oregon 



Action Agenda 

Thank you very much. 
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Business Perspectives on Using Data 
for Improvement and Accountability 

Martha Darling 
Washington Business Roundtable 

I am delighted to join you here today. I am pleased also to recognize a 
face or two in the crowd from the State of Washington, We've had quite a bit 
of contact with the education community in Washington, and it is obviously 
from that perspective that my remarks will come. Just in the interest of fiir 
disclosure, I am a Vice President of Seattle First National Bank and have been 
on long-term loan from the Bank to the Washington Roundtable for almost two 
years to head up the education study. 

I'll tell you a little bit more about the Roundtable itself. Dean Mosier 
alluded to its two and a half years of existence and the fact that education 
was one of the first two issues that the Roundtable took up. The Roundtable 
is made up of 34 chief executive officers in the State. It fluctuates up and 
down; if you sell your company, you're out. (It's a psetty harsh set of 
rules. Presumably there are counterbalancing factors from selling one's 
company under certain t nditions. There are also two citizen members — the 
sort of distinguished citizens of the State who have been involved in all 
kinds of things. 

The goal of the Roundtable is to apply the knowledge, creativity and 
leadership resources of the companies to long-term problems and challenges 
facing the State of Washington. Now that sounds fine. How is it applied? 
The Roundtable took on two items at the start and intends to keep its agenda 
limited at any given time. Those two were state fiscal policy — debt 
management, pension funding. Then there was education. T. Wilson of the 
Boeing Company was the first chairman of the education committee; he is now 
Chairman of the Roundtable. After some months, it was determined that people 
who had full-time jobs were not going to be able to handle on a part-time 
basis the magnitude of the task looming ahead. This realization led to a 
recommendation to the Roundtable that a major commitment of resources be made 
by the companies. Out of that came Seafirst's long-term loan of me, plus a 
commitment from companies to loan executives full-time for periods that have 
been averaging four to five months, to work on the education study. 

Our approach was to identify priority issues on the basis of talking with 
a lot of people from education and froa the larger community. Those took the 
form of questions such as: Is the curriculum adequate? If not, what should 
be done? We asked similar kii^s of questions about student testing, teacher 
compensation, teacher evaluation, teacher preparation, the role of the 
principal, early childhood education programs, and so forth. In each case we 
asked, are things adequate? If not, what should be done? 

Now about the topic at hand, which in my particular case is the business 
perspective on using data for improvement and accountability. Educational 
quality and concerns about improvement have been traditionally defined and 
measured by the character of the inputs — per pupil expenditure, teacher 
preparation, class size— we've all seen these kinds of charts. We have also 
looked at processes — planning systems in place in the schools, training for 
personnel in schools and so on. Now — and the business community certainly has 



been a part of this new thrust— there is a greater focus on outputs, student 
performance on achievement tests perhaps being the roost obvious. The question 
of how to produce higher student achievement has now somewhat overshadowed the 
question of how to be a better teacher. 

These are two different perspectives on the educational enterprise. I 
think it is very important to see the shift of attention to outcomes, tusiness 
and the outside community are looking at education more closely and as new 
monies are sought. The fundamental question regarding use of the new money is 
what are you getting for what you spend? To say that there is better teaching 
going on without being able to cite the juoduct (higher student achievement or 
some other outcome measures) is not going to get you very far. As I say, this 
is a natural kind of thought on the part of business, and it does not exclude 
a concern with inputs and processes. In business— and this is something that 
you all appreciate, I am sure— a lot of money is put into inservice training, 
especially the higher \ you go. It is shameful that education does not 
invest significantly in a process presumably designed to make better 
teachers. The Roundtable has not ignored that issue, but, after you give 
inservice training, do you hav*. any measures to tell you that you have gotten 
anything for it? you know better than I the amount of Mickey House stuff that 
goes on under the rubric of "inservico," and under the rubric of "continuing 
education credits". 7'<ere are worthwhile programs as well. Until we can 
start sorting out the one from the other, I think it is going to continue to 
be very difficult to get public monies where they ought to be going. 

When the "oundtable approached the education topic, we began by making the 
naive assumption that certainly there would be some descriptive brochure 
telling Uw about the governance and the financing of the educational system in 
the State of Washington. But, did it exist? Not at all. The result of that 
particular discovery was a project that required a wall roughl "s size 
(indicating the wail) at the Boeing Company to chart different /ers and the 
lines that connect them. It took a wall to accommodate the edu Ion system. 
When people saw the wall, they could say, "Well, no, that's not te right; 

it really works line this " And the little name cards and t..e lines would 

be moved. More than one person— especially those from the 
legislature— remarked, "row this would really be useful to have or paper. 
Coming to Boeing to S.ook at this wall every once in a while is not the most 
efficient way to make use of this information." 

The result— affectionately known as the "Off the Wall" project— has been 
a booklet called Educational Gove r nance and Finance in the State of Washington 
put out by the Roundtable. Recommended for bedtime reading, it is a major 
early Roundtable contribution— describing these systems for the firs*, time. 

I recite this experience to highlight two major problems relating to data 
about education in the State of Washignton— and I suspect elsewhere, from what 
I have heard. First, there is much daca but little information. The second 
one derives from that fact that we have a highly decentralized system, as many 
states do, with a strong tradition of local control. Forget the fact that the 
State of Washington now pays 78 percent of the bill; the local control 
tradition is still hallowed. That means that a lot of information exists at 
the district level which is not centrally available, or, to the degree that it 
is collected centrally, it is not comparable. Definitions vary this way and 
that way. Some would say that there is protection in this lack of clarity. 
But I think times have changed as the chief state school officers have 



recognised, with their study directed at identifying valid indicators of 
school effectiveness, and developing greater standardization of what is being 
measured in the schools. 



At any rate, lacking much needed information, we resorted to visiting a 
lot of school districts. In the course of our study of 15 priority K-12 
issues, our loaned executive analysts visited no fewer than one-third of tht 
State 1 s 298 school districts. This nap (displaying a map of Washington) shows 
the districts in every part of the State where we visited. It was very 
enriching for the study to get the real flavor of what is going on in the 
classrooms, see the variations that make it ridiculous to generalize across 
districts, but also to note some of the kinds of conditions that do make it 
possible to generalize across districts. 

We also did a teacher survey. In cooperation with the Washington 
Education Association 'WEA) we mailed the survey form to 5,000 teachers in the 
State, one out of every seven classroom teachers. WEA helped us word our 
questions so that they would be more meaningful to teachers, and this was very 
helpful* We had a return of over 2,000, which is about double what anybody 
expected. The respondents provided a large amount of information on items 
ranging from working conditions — which as you might expect were deemed pretty 
horrible—to attitudes of students and their preparedness to learn, attitudes 
of parrents, views of school administrators, availability of inservice, summer 
work, academic course work, and so on. We were able to stratify that 
information by district size, years* in teaching, and a number of other 
features. Finally, we did quite a bit of souxcing in other states, talking on 
the phone with education leaders and political decision makers, and talking 
with some of the national associations about what they knew about what was 
going on elsewhere. 

With all of that as preliminary, let me get on to the data we collected 
and what we aid with the information. As you can imagine, the attitude in the 
Washington education community about thtj prospect of this business group 
coming in and doing a green eyeshade job on education was not exactly posiive 
at the start. There was suspicion that we would say, "If you cut budgets here 
and cut out these admins trative positions there, education will improve. " As 
things went along, and as w<* adopted the method we did, I think we gained 
acceptance. While our findings were not always in accord with what educators 
thought was appropriate, people did seem to understand that we were not coming 
in with our minds made up, we were not seeking only to validate some notions 
we already had. Instead, we were sincerely interested in trying to understand 
what the enterprise of education was about. That is in marked contrast to the 
approach taken by some business groups in other states. I think you all know 
that Ross Perot and th*» government of the State of Texas were seen as pretty 
much rolling over the education community in instituting some of their 
reforms. California and Minnesota hired outside consultants, and you know how 
popular consultants are. 



On to our issues. The f*' ^ne I will talh e.bout briefly is student 
testing. There are actually two issues here. One was statewide student 
achievement testing on a tstandardized basis. The other was minimum competency 
testing* Until this year, the only standardized student achievement test 
administered statewide was at fourth grade level. It was on the basis of that 
fourth grade test that the State Superintendent of Public Instruction would 
boast periodically about the quality of Washington's education system—that 




test combined with the SATs, which also indicated that Washington was doing 
quite well nationally. As you night have already concluded , there are 
problems with both of those measures, and it did not take u great deal of 
grounding in education to decide what those problems were. For one thing, the 
fourth grade test given in October of the fourth grade is really a test of 
third grade achievement and it is hardly a measure of 12 years worth of 
education. On the contrary, there were some strong suspicions that the 
problems really began in the lave elementary grades, got worse in the niddle 
school, and, with a 25 percent dropout rate in high school, maybe culminated 
there. Fourth grade did not seem to be quite adequate as a measure of 
anything in particular. 

In 1984 the legislature instituted an eighth grade tes\ and the Roundtable 
advocated the addition of a tentj grade test. We believed that these three 
measures would give some indication through time of what was going on in the 
school system, and would also yield information that could be used for 
diagnostic purposes by the school districts. We agree that test results often 
are misused. For example, when you compare the relatively prosperous district 
with the relatively poorer district, that relatively prosperous district is 
always going to look better. That does not mean that they are doing what they 
should be doing with those kids. For that reason we recommended that test 
scores be reported in two ways: one is the raw scores, and the other is 
demographically adjusted, so that you can compare like districts. 

The legislature passed the tenth grade test bill this last Spring without 
the demographic adjustment, but we will continue to be visiting with there 
about that. 

Now about the SATs. In the state of Washington, 16 percent of our high 
school students take the SAT. When you take a look at the national rankings, 
you see that for almost all of the states that are high in the rankings, only 
three, four, six, seven, ten percent of the kids are taking the tests. It 
does not take much logic to conclude that there is soae selectivity going on 
as to who is taking those tests. . .perhaps the students who are aspiring to go 
out of state, to the Ivy League? Anything that usee the SAT on that basis, 
such as the Secretary's Wall Chart, is ridiculous, misleading, and frankly 
gets to the point of simply being dishonest. We tossed out the SAT test after 
we adjusted it with our Washington Pre-College Test results, which a larger 
percentage of kids take. On that basis Washington fell below the national 
average. The State Superintendent's office has not given any publicity to 
that. 

All we are saying is that if you intend for test scores to be useful 
feedback devices, you must choose and use them carefully. Which moves right 
along to the minimum competency test. If there is one area that one would 
hr*e guaranteed that business would take a positive position, this is it, 
right? Minimum competency: You set a standard; you get over it or you 
don't. It was the first real test for the Roundtable, in the Spring of 1984. 
As a result of our study, the Roundtable took a position i . opposition to a 
state minimum competency test for high school graduation. This position was 
an experience base in Washington and in other states. From our review of 
minimum competency tests nationally, we found that after they had been in 
operation for a while, only one to two percent of the kids actually failed to 
graduate from high school on the basis of test failure. We said, "This is an 
interesting coincidence of statistics. 11 We looked more closely and found that 
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this remarkable statistical regularity was achieved by dummying down the 
test. Political heat said you could not tolerate more than about that level 
of failure. If you needed to to reduce your test to a tenth grade level to 
achieve it, you did so. If you had to reduce it to an eighth grade level, you 
did so. 

At this point we asked, "What kind of benefit is minimum competency 
testing if it so distorts the activities of many school systems in order to 
get kids over that hurdle?* We were particularly concerned with the shifting 
of resources from higher order skills development to remedial courses because 
of legal requirements. We looked at the states which have a MCT requirement. 
They tend to be states with relatively lower educational attainments. We 
said, "This is not for us." To achieve the sorts of goals that minimum 
competency tests are supposed to be associated with, we thought it was better 
to concentrate on our records on high school curriculum, the standardized 
achievement testing, and remedial course funding, combined with some 
preventive measures in the early childhood area. We determined, in effect, 
that a minimum competency test was no indicator of any improvement, we also 
looked at minimum competency testing as it existed in a few Washington school 
districts and concluded that many are on very shaky grounds. So much for 
minimum Cv petency testing. 

Another area that I will mention briefly is teacher compensation. Teacher 
compensation is determined at the state level, and the State of Washington 
uses a schedule based on seniority and education — two false proxies, not 
always accurate in my opinion, for excellence in teaching ability. First, we 
looked at career compar isons with industry, we asked how * teacher's 
experience on a dollars-per-day basis worked compared with somebody in 
industry with a comparable initial degree over a period of 20 years. There 
were chree significant problems. Entry pay was too low. At 15 years the 
teacher pay topped off — plateaued. Finally, very good teachers and so-so 
teachers at similar levels of experience and education received the same pay; 
performance was not a factor. The overall result was that teachers in general 
were reasonably well paid compared with workers in industry, but good teachers 
were underpaid. 

The Round table looked broadly at different approaches to teacher 
compensation, including systems in place or under implementation in other 
states. After studying various proposals, the Roundtable developed a career 
ladder/mentor teacher system which we recommended to the legislature. While 
the idea was discussed and debated extensively in the state capitol and 
elsewhere, it represented such a change from the current system that action 
was deferred in favor of more study. The Roundtable will be back on the issue 
of performance-based pay for teachers, an idea whose time has come, in our 
opinion. 

The final Roundtable recommendation I will highlight addresses "time on 
task", often considered in terms of the length of the school day and year. 
Our review of experiences with longer hours of schooling in other states and 
nations, combined with a look at how existing time is used, led us to conclude 
that the economises "law of diminishing marginal utility" was clearly 
relevant. Many of you are more familiar than I with the Academic Learning 
Time research, one of whose basic messages is that we are perhaps only 
one-third "learning efficient" in our use of the current school day. While 



being on task 100 percent of the time is unrealistic— and counter to human 
nature— an increase to 50 percent would bring huge benefits to both students 
and teachers, and at very little additional cost. 

We then turned to several Washington school districts which had undertaken 
programs aimed at increasing academic learning time. On the basis of our 
visits, interviews and observations, we recommended a statewide academic 
efficiency training program. The program first called for a self-diagnosis 
phase, during which teachers and administrators determine the time lost from 
learning activities, and the policies at the district, school, and classroom 
level which contribute to or cause the loss. A second phase involves 
classroom management training for a limited number of teachers per school, who 
would then return and train their colleagues in techniques to increase 
academic learning time* The good news is that the legislature passed our 
bill, recognizing, we believe, both its tremendous potential benefits and its 
small cost. The state is now moving toward implementation, which we intend to 
monitor closely. 

I cannot close my review of our experience in "using data for improvement 
and accountability 19 without a war story — a live example of data ignored* 

The issue was class size. In last spring's legislature, the question 
actually began as a proposal to get more dollars to school, districts. Under 
Washington's Basic Education Act, funding is provided on the assumption of 50 
certificated staff per 1000 students; the original proposal from the State 
Superintendent was to increase the ratio over five years to 55:1000. In view 
of the cost projections on the one hand and the state 1 s dwindling resources on 
the other, the proposal was scaled back several times and was eventually 
transformed into a program to reduce class size for grades kindergarten 
through three and for high school vocational e .ucation. 

No appeal to research was ever made by proponents, only tc intuitive 
feelings on the part of most parents, teachers, and school administrators that 
any class size reduction makes things better. Although the research on class 
size effects is reasonably clear on the benefits of reducing class size below 
35, and indicates that classes of no more than 15 also demonstrate prominant 
gains, the degree of benefit resulting from incremental reductions in class 
size between 35 and 15 is significantly reduced. Costs, meanwhile, in the 
form of classroom space and teachers, are all too obvious. As a consequence 
of our review of the research, the Roundtable suggested that before embarking 
on the expenditure of large amounts of money, the state should fund a class 
size pilot to determine the student outcome effects of different class sizes, 
ranging frcat 30 down to 10. The legislators, however, aiued and abetted by 
school people and parents, were not interested. More than 
$9,000,000 — two-thirds of the new money provided to K-12 education— went for 
reductions in class size of questionable benefit. 

Before finishing, a few words placing our work in perspective. Absent 
from our approach in Washington is a more comprehensive approach to K-12 
education which could tell the Governor, legislature, the State 
Superintendents office, business, parents, and citizens whether we are 
getting the education results we are paying for— and in Washington, the state 
provides 78 percent of the costs of local education. Perhaps we, as a state 
and as a region, are a little self-satisfied. We come off pretty well in 
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national comparisons. In contrast, I would cite South Carolina, which has one 
of the most elaborated systems of accountability in the nation following 
passage of their comprehensive Educational Improvement Act of 1984. A 
Division of Public Accountability in the state Department of Education was 
created with a three-year life to answer that fundamental question, "Are we 
getting our money's worth in improvement?" They have created a system to 
evaluate quality In all school districts through monitoring selected outputs 
and inputs. Data collection is a key element. While our needs to improve our 
educational system may appear less than South Carolina's, their approach to 
data collection and use iuay hold lessons for us all. 

Thank you for allowing me to share my thoughts with you today, and good 
wishes on a productive meeting. 
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State Perspectives on Using Data 
for Improvement and Accountability 

Don Egge 
Oregon Department of Education 



It is nice to be among friends even though they may not be your own. 
I arc anxious to hear what I am going to say because I found myself 
wanting to change some of my planned comments to include responses to 
things I have heard this morning. Before I begin, I howeve*, have 
something important to say. 

Bob Rath mentioned the educational reform initiatives across the 
country, and Oregon, as other states, has been involved with that 
movement., About a year and a half ago, the State Board of Education 
adopted what we call the Oregon Action Plan for Excellence. That plan, 
as I understand it, has three major theme* running through it: 

1. If you want excellence and improvement, change efforts need to 
take place at the classroom and school level. 

2. Schooling should neither be totally state nor locally 
controlled; rather, each has a proper role to play, and these 
need to be identified. The Plan calls for the state to define 
what all students are to learn and to measure and it. The local 
district is to define the comprehensive cirriculum and determine 
the means to achieving both. 

3. If there is going to he improvement and excellence at the local 
level, then local schools need to develop and maintain the 
capacity for self-renewal. All other agencies concerned with 
the educational system have a responsibility to help the schools 
build that capacity, and the state has a special obligation to 
establish a support system to help them to do so. 

This conference, therefore, will surely help us to understand how to 
measure and monitor implementation of our reform initiatives as well as 
changes in local and state performance. 

My assignment is to talk about state perspectives. Rather than 
discussing the same issues that my colleagues will be sharing with you 
later, I will offer some reflections I have had from my recent six-week 
internship in Washington, D.C. with the indicators project at the 
National Center for Educational Statistics. 

Educational policy making takes place at all levels — the federal 
level, state, school district, and school. In the absence of clear 
policies established by these groups, you can often find individual 
teachers setting and implementing policy, whether we participate in the 
policy making process as advisors, deciders, implementors, or clients, we 
have an interest in data and information. We have a tendency to think 
about policy making as something that takes place behind closed doors in 
smoke-filled rooms, and in fact, policy making can be a very messy 



process. Indeed, some say that there are two things you never want to 
watch in the making — laws and sausage. While not all policy making is as 
messy as I have been suggesting, we are seeing policy making in a more 
open setting today, and this can be cumbersome. There is more 
interaction among levels and with different groups than ever before, and 
individual persons are now able to affect policy at a variety of levels. 

This is an interesting time because, as was suggested this morning, 
we are said to be in the middle of a paradigm shift in our society 
between the Industrial and the Information Ages. This means that people 
and organizations have more and more knowledge about data — how to use it, 
where to get it, and how to apply it to effect the results that they 
want. When yon mix that with the consumer movement of the last 15 years, 
the effect is that different interest groups are vying for access to the 
most information so as to sway the public and to sway the policy makers 
and their direction. 

How many of you know where Grays Harbor City, Washington is located? 
My mother lives in Grays Harbor City, a thriving community at the turn of 
the century, now there are maybe 20 homes in that area. It is just 
outside the community of Hoquiam where I was raised and educated. 
Hoquiam is an Indian name which means "hungry for wood. " Now if you can 
picture the Hoquiam River coming down and emptying into Grays Harbor back 
before the turn of the century, logs and branches were piled up on the 
delta. The Indians observed the situation and said "hungry £or wood." 
So Hoquiam became the name of that community. When the settlers arrived, 
they found not only a pile of wood, but a delta full of water. They 
hauled the wood off to sawmills, took the sawdust and filled the swamps, 
so that a community could be built on the site. Supply and demand were 
reconciled to solve a major problem. The transformation of the wood into 
the more useful sawdust has an analog in the translation of data into 
information which is meaningful and useful for school improvement 
decisions. 

In recent years many large-scale studies have compared nations, 
states, districts, schools and even classrooms. A lot of the publicity 
has been given to these investigations, and they have produced valuable 
information for use in schools and classrooms. The findings from these 
studies have caused state departments of education to take a greater 
interest in defining which dimensions of schooling should be measured, 
mointored over time and reported to be clear and public about educational 
performance. These are important concerns for several reasons. The 
publication of data has brought education to the attention of the public 
and has made it an issue of enduring concern. Presenting accurate data 
may increasingly aid local, state and national policymakers in 
understanding the consequences of policy change and aid them in 
implementing policies once they are adopted. Finally, sharing 
educational data makes educators more accountable to the public. 
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So, as the states work to influence the collection and reporting of 
data, local school district personnel ought to be concerned about the 
ways this will affect them. Just as the states are struggling to develop 
a clear connection with the federal government, so local districts in 
many states are trying to establish some kind of relationship with their 
own state. This is the situation in Oregon at the present time. At the 
stace level we are working to produce the right kind of data and 
information and report these in a meaningful, positive kind of way, 
without negatively affecting the many excellent assessment systems that 
have been developing in local school districts in recent years. 

The Council of Chief State School Officers recently established a 
center on assessment and evaluation , appointed a director, and is 
beginning to define terms, establish criteria for selection of 
indicators, select indicators and establish data bases to help states in 
their work. The Council is trying, I believe, to establish a kind of 
infrastructure, or at least a connection between state and federal 
levels. The Center 1 s work should be helpful to us in our individual 
states and even in local school districts. There are also a variety of 
other national, state and local activities underway across the nation in 
selecting, measuring and reporting indicators. 

Until recently we in the educational system have not made fully 
effective use of data and information, either for our own management 
purposes or for communicating with the public. One reason for this is 
that we have not had very good tools for the collection, analysis and 
interpretation of data. There are, of course, many complexities in the 
educational process which make it difficult to have very simple, direct 
kinds of measures. However, many improvements are being made in tl*e wasy 
we specify and measure educational effectiveness. With these kinds of 
activities comes the increased visibility I mentioned earlier. Whenever 
you start recording a variety of data, you suddenly make your 
institution — whether it's a school, a school program, a district or even 
a state— very visible. That can produce both positive and negative 
responses. One potential negative effect—and one *hose of us here have 
probably all experienced—is that when we are open and honest and direct 
in revealing a weakness in our system, our detractors sometimes sieze 
upon this and criticize us harshly, we need to learn to anticipate and 
cope with these situations. Perhaps the business community can be 
helpful in explaining to the community at larg* that every enterprise 
must monitor key indicators of condition and performance in order to make 
changes and adjustments which bring improvement. 

One of our difficulties in monitoring and making needed adjustments 
is that schools have very diffuse goals. If you only measure the basic 
skills at fourth grade, you 1 re only just touching on the tip of the 
iceberg. We have, in addition to instruction, other functional 
responsibilites such as socialization, custody control, evaluation, and 
certification. And although we are accountable for those functions, we 
frequently lack good data on how well we are doing in these 
non instructional areas. Thus, we have a major responsibility as policy 
makers or advisors to policy makers to select indicators of performance 
which will be representative of our systems. 
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Of course, legislatures, state superintendents, state boards and 
local admi istators and boards have always held up sets of indicators and 
though ofent randomly selected they have been used to defend positons. 
They night be simple things like class size, costs per pupiJ , student 
scores, achievement tests, and dropout rates. So it is not that we 
haven't had some measures; it's just that they have not been a cohesive 
group of indicators, and they have not covered all of the major areas of 
activity for which the educational system is held accountable. In 
choosing additional indicators, however, we must reflect the real 
education system as close as we can without specifying so many indicators 
chat the monitoring process becomes unwieldy. 

When we select indicators, we have to think carefully and in an 
organized way. To illustrate this process, let me suggest a simple model 
which represents a framework for looking at indicators actually used by 
various states though it coulbe be applied at the local level. It covers 
system performance by organizing indicators around resources, process amd 
outcomes as well as variables for measuring context. I developed the 
framework and gathered these data as part of my work experience in 
Washington. (See the Attachment to this paper.) 

The first set of indicators are outcomes . The first group of outcome 
inoicators covers academic breadth and depth of student achievement. The 
second is concerned with student progress and transitions, either in the 
school system or as students move outside the system. Third, ve can look 
at equity issues of access, participation, and the impact on special 
populations such as the blind, and the learning disabled. We can also 
look at other program and services, and finally at indicators of 
community confidence. 

Let's look at some of these relative to academic breath. Some states 
are using test scores on high school exit examinations as an evaluation 
across the entire system. From the Secretary's wall Chart, we find a new 
indicator, "Percentage of State Gains to meet the President's Challenge 
on SAT Score Change. 9 Other indicators of Academic Depth, might include 
SAT achievement scores or state achievement test scores. 

For student progress tnere are many potential indicators — for 
example, the percentage of students ready for first grade. Peformance 
can be monitored overtime to determine whether readiness programs, or 
kindergarten programs are effective. Percentage of grade level repeaters 
is a transition indie a tot. Looking et school comoletion, secondary 
attainment might include grade level attainment, so that instead of just 
talking about dropouts, you could measure the grade levels actually 
attained by student. The educational aspirations of senior students 
would not necessarily be very hard data, but the indicators might be 
useful to see how your guidance program might be influencing student 
expectations. With regard to equity, useful indicators might include the 
percentage of special education students mainstreamed, if mainstreaming 
is a major goal. Community confidence indicators include parent and 
student satisfaction rates. 
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Let's turn our attention, to process indicators . Included are 
program content, instructional climate, support to instruction, and 
program productivity. One of our problems with process is that we really 
have a very limited understanding of our technology. This is something 
that is confounding to those who have r^ver been in the classroom, 
because when you are in an elementary classroom with 25 youngsters who 
have mental ages ranging from 7 to 15, you have 10 to 12 content areas to 
cover, and ycu have runny noses and students with the flu, and you are 
constantly making adjustments and decisions so that, what actually 
happens is a complex process based on pans, experience judgment. 

It is almost as if we have a "black box model" which is used in many 
areas of research and development. According to this model, we measure 
input and output while establishing some hypotheses about what it takes 
to produce the desired results, we have all learned, however, tnat there 
is more than one way to skin a cat, or deliver results, and because of 
individual student learning modalities and rates, we must be able to 
apply a whole variety of technologies to be able to produce the results 
we want. Given that kind of a model, we can begin to think about a way 
of looking at the indicators we select. 

Some important measures of breadth of content are number of programs 
available, number of student slots available, and number of classes 
taught by teachers out of the field. We might choose to measure 
engagement, the number of students served in various kinds of programs; 
or challenge, the percentage of students completing upper level courses, 
or trends in difficulty in courses chosen by high school students. 
School climate variables include such things as satisfaction rates and 
staff perception of the threat of crime and violence. For support to 
instruction, we can look, for example, at the percentage of classrooms 
with access to computers; the professional and nonprofessional staff 
ratio; and the number of improvement activities. Productivity 
improvement measures include ratios such as program credits earned 
divided by direct program costs, or for the entire school, the number of 
credits earned by students divided by the full-time faculty, to determine 
whether you are using flexible means of granting credit. If you are 
looking at a support program like guidance and counseling, you might want 
to monitor the time available for counseling and the time actually spe it 
with students. 

Next, we turn our attention to the resource indicators — human, 
fiscal, and community. Looking at human resources, we 9 11 be concerned 
with indicators of the quantity and quality of teaching force. The 
fiscal effort is something we hav? often looked at; here we measure it 
not only from the community base, but also the impact on the individual 
teachers, for example. When reviewing expenditures, we have typically 
looked at the instructional cost per pupil, but there are several other 
things we might choose to measure. Finally within the area of resources, 
we might look at the effort of community organizations, the kind of 
contributions that they make. If a part of our model is to set a goal of 
increased community participation, then we can monitor that expectation 
in several ways. 



The second major category with which we aie concerned is that of the 
context variables that influence the system. The first of these are 
demographic ones. Looking at these, I want to point out that factors 
such as attendance and dropout rates serve to illustrate the relationship 
between context and outcome variables, with dropouts and attendance, the 
school can influence student decisions, but for the most part they are 
decisions of the family and individual student and these context. 
Looking at holding power, one need in our state is to move away from 
looking at a single dropout rate, in favor of monitoring a variety of 
indicator*. We've been moving in this direction, looking at other 
indicacors such as graduation rates, annual attrition rate, and so on, 
which helps the public to get a more rounded picture than is possible by 
presenting the dropout rate alone. If student pass away, go to c 
community college, or get a GED, they are counted in the dropout 
statistics, which is extremely misleading. 

Other contextual variables to consider are those having to do with 
policies and mandates which affect schools and districts, and those 
having to do with economic conditions. We do not have time to discuss 
these today, but they are obviously major influences of educational 
decisions and operations. 

The indicators of effectiveness presented in this compilation do not 
comprise a comprehensive set. It is important that whatever indicators 
you use they are defensible from the standpoint < : how well they reflect 
the real world in your school, your program, your district, or your 
state. 

My major point this morning is that we need to make better use of 
available data and create better information. Selecting indicators must 
be based on the goals you have for your system. We need to be aware that 
gathering and sharing more data and information can be expected to 
increase our exposure and visibility, but we neea not fear this. 
Instead, knowing that we will be under greater scrutiny ought to motivate 
us to take steps to insure that we have accurate data and useful 
information to apply to policy making at every level. 
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APPENDIX A 



A FRAMEWORK FOR SELECTING INDICATORS TO MONITOR THE CONDITION 
AND PERFORMANCE OF A STATE EDUCATIONAL SYSTEM 

I. INDICATORS OF SYSTEM STATUS, A STRUCTURE 

A. OUTCOMES 

1.0 Student Achievement 

1.1 Academl: breadth 1n basic skills, general knowledge and aptitudes 

1.2 Academic depth 1n subjects 

1.3 Functional literacy 

1.4 Extracurricular activities 

2.0 Student Progress 

2.1 Entry to school 

2.2 In-School 

2.3 School completion 

2.4 Postsecondary transitions 

3.0 Equity 

3.1 Access to programs for special needs 

3.2 Participation 1n special programs 

3.3 Impact on special populations 

4 0 Other Program Services 

4.1 Student services 

4.2 Media services 

4.3 Auxiliary services 

5.0 Community Confidence 

5.1 Community satisfaction 

5.2 Parent satisfaction 

5.3 Student satisfaction 
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I&K5IFIWIIW 5TTUCIURE WITH INDICATORS OR INDICATOR 

hHtoS (RECENT NATIONAL ACTIVITY KEY: SECRETARY'S 
mil CHART (SWCL INDICATORS: STATUS OF TRENOS (^ES). 
pmfl. mim \* $BB& CONSORT! JW (ELC) 

^OUTCOME INDICATES 

s&I.O student Achievement 

■ 1.1 Medemlc Breadth 

a. Achievement scores on standardized tests 
reported froM local testing programs 

b. Test scores on high school exit examination 

c. Per f ormance of students over time Mho 
wtlclpated fn the special progress 

d. Achievement scores on standardised tests 
administered statewide at selected grades 

e. Perforate of college preparatory students 
compared to that of nonpreparatory students 
on SAT and ACT 

f. State ranks on score chrnge on ACT (SMC) 

g. State ranks on score change on SAT (SWC) 

h. Percentage of state's gain to meet President's 
challenge on SAT/ACT score change (SWC) 

1. Achievement of 9, 13, 17 year-olds In R, , 
Scl (SUC, NOES) 

j. College bound student achievement as Measured 
by SAT, ACT, PSAT (SWC) (NCES) 

k. International comparison of math achievement 
at 8th and 12th grades (NCES) 



to 
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1 . Reading by race, ethnicity 

Percent mastering the vocabulary necessary 
for an Individual to function with a reasa 
i jtonomy in our complex society 
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SL 

SL 
SL 
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NS 
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TENTATIVE STATE-LliVEL EXAMPLES 



CA 



CO 



CT 



FL 



KY 



HI 



«S 



NV 



NY 



OH 



OR 



SC 



TX 



UT 



WA 



WV 



on*], (S) State, (L) Local levels 
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, , CLASSIFICAT ION SIHUC I WE WW I nMMMS OR I No I CA T » 

. J^AREAS («CEMT NATIONAL ACTIVITY KEY: SECRETARY'S 
i . MALI C.«T (SMCK INDICATORS: STATUS OF TRENDS (NCES), 
1 EOUCATION LEADER? ^QNSORTjUH (£LC) ' 

2.3 School Completion 

a. High school graduation rates (NCES) (SWC) 

b. Secondary education attainment 

c. Functional competency testing adults 

d. Percentage points to be gained to meet Presi- 
dent's challenge on graduation rate (SMC) 

e. Number of GEDs 

f. Number of adults passing GEO test 

2.4 Postsecondary Transitions 

a. Activities of graduates the first year after 
high school (NCES) 

b. Remedial course enrollment of college freshmen 

(NCES) 

c. College success of the state's high school 
graduates in state institutions of higher 
education 

d. Increased percentage of students completing 
vocational-technical programs and working 
in the field of their training 

e. Freshman year grades in postsecondary programs 

f . Increased percent of students receiving 
high school diplomas on schedule 

g. Educational level aspirations of senior 
students 

h. College entrants 

i. Percentage of vocational students placed 

j. Postsecondary success in reaching destination 

k. Percental of high school students with 
early postsecondary entry 
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TENTATIVE STATE-LEVEL EXAMPLES 
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UAUlrlCMlOH SIMJCIIH: U1IH 1N01CATU6 W INDICATOW 
WgUS (RECENT NATIONAL ACTIVITY KEY: SECRETARY'S 
^^.S ng^T?^: STATUS OF TRENOS (NCES). 
'BDCCATIOM LEADERS CONSORTIUM (ELC) 



4.0 OWwr Prograr 'Jervlces 

4.1 Student Servl ces 

4.2 ftedU Services 

4.3 Auxiliary Services 
5.0 Omaunlty Confidence 

5.1 Canunlty Satisfaction 

a. Commlty satisfaction rates 

b. Employer ranking of satisfaction with work 
performnce of graduates of the state's 
public school program 

5.2 Parent Satisfaction 

a. Parent satisfaction rates 

5.3 Student Satisfaction 

a. Expressed adequacy of schooling by students 



b. Percentage of students satisfied wtth 
achi ev eme n t of next step beyond school 
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TENTATIVE STATE-LEVEL EXAMPLES 
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B. PROCESS '■"'?'< 



1.0 Program Content 

1.1 Utility (the relevance of the written curriculum for graduates) 

1.2 Breadth of program coverage 

2.0 Instruction 

2.1 Opportunity for quality experience 

2.2 Engagement 1n academic pursuits 

2.3 Challenging students to participate 1n rigorous programs 
3.0 Climate 

3.1 Instructional leadership 

3.2 Achievement expectations 

3.3 Systematic and personalized Instruction 

3.4 Staff morale 

3.5 Safe and orderly environment 

3.6 Focus on task 

3.7 Student and staff Interaction 

4.0 Support to Instruction 

4.1 Curriculum materials 

4.2 Supervision and assistance 

4.3 Improvement activities 

5.0 Program Productivity 

5.1 Instructional programs 

5.2 Support programs 
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| raOCESS INDICATORS 
- 1.0 Content 

f 1.1 utility (relevance) 

a. Graduate rating of school orientation used 
in post-school experience 

1.2 Breadth 

a. Mater and percentage of students served in 
special programs 

b. The increase in mmfcer of students in drop out 
prevention program 

c. Jfcmtwr of vocational programs available 
to students 

2.0 Instruction 

2.1 Opportunity 

a. Nufter of classes taught by teachers 
out-of-field 



00 



b. Pupil/teacher ratio in selected subjects 
for grades 1 through 3 and 7 through 12 

c. PTE Studenfc/FTE Faculty Ratio (SWC) 

d. Length of school day in periods of instruction 

e. Length of school year in days taught 

f . Enrollment by type of curriculum 

g. Compensatory education program in the state 

(SIC) 

h. Average class siro (ELC) (MCES) 

1. Class size by program 

j. Jtaber of separate teacher "course 
preparations" during the day 
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3.0 



h. Enrollment in selected academic courses 

i. Percent of high school students with personal 
carter plans on file 

j. Bean ytarv of study in selected subjects 

k. Stat* required Carnegie units by subject (NCES 

CliMte 

3.1 Instructional Leadership 

3.2 Achi e ve me nt Expectations 

3.3 Systematic and Personalized Instruction 

3.4 Staff Morale 
a. School personnel satisfaction rates (ELC) 

3.5 Safe and Orderly Environment 

a. Degree of staff satisfaction with approaches 
to discipline 

b. Disciplinary actions 

c. Vandalism rates 

d. Vandalism costs 

s. Student perceptions of the threat of 
crime and violence 

f . Staff perception of the threat of crime 
and violence 

g. Comwnity perceptions of the threat of 
crime and violence 

3.6 Focus on Task 
a. The number of classroom Interruptions 

3.7 Student ano Staff Interaction 
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5.2 Support Programs 

a. Total student contact hours (by counselor) 
during alum t\m period divided by total 
available counselor hours 

b. Total student hours of direct nedls canter 
contact per year divided by annual personnel 
costs of aedta center staff 

c. Nwber of mm Is of given standards served 
divided by total food service costs 
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C. RESOURCES 




r 


1.0 


Human 








1.1 Quantity 

1.2 Quality 

1.3 Enhancing staff capacity (extending Impact of resources) 






2.0 


Fiscal 

2.1 Effort (commitment to sunport school programs) 

2.2 Expenditures (actual payments for school) 
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3.0 


Community 
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3.1 Effort of community organizations 

3.2 Citizen and parent participation 

3.3 School status 
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«|;?^0 Humt« 
I 1.1 Quantity 

a. Nwfcer of teachers par 100 pupils (alterable 
at the district level) 

b. Average of regular faculty present each day 
(indirectly alterable by district policies) 

(ELC) 

c. Nujfcers of staff employed 

d. Teacher supply and demand (NCES) (ELC) 

e. Pupil teacher ratio (NCES) (SMC) 

f. Pupil teacher ratio by program 

g. Pupil staff ratio (SMC) (ELC) 

1.2 Quality 

a. Quality of teaching force (verbal SAT scores) 
S (NCES) 

b. Quality of teacher fore* (verbal GRE scores) 

c. Percentage of teachers with advanced degrees 

d. Certification rank of teachers 

e. Percentage of minority staff ami teuchers 

f. Teacher retention rate 

1.3 Extending Staff Capacities 

a. Span of administrative responsibility (ELC) 

b. Professional/nonprofessional ratio 
2.0 Fiscal 

2.1 Effort 

a. Percentage of revenue from local sources 

b. Public school fiscal effort (NCES) 
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3*2 Citizen Participation 

a* Nurter and percentage of parents and 
comunity members serving the school 



(Etc) 



b. Estimated hours of volunteer service during 
the year (total and as a percentage of AON) 

c. Interagency cooperative agreements 
3.3 School Status 

a. Percentage of schools holding regional and 
state accreditation 
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II. CONTEXT VARIABLES 



A. OEHOGRAPHIC 



1.0 Students 

1.1 Characteristics 

1.2 Enrollment 

1.3 Attendance 

1.4 Holding Power 

1.5 Special Needs 

2.0 Community 

2.1 Characteristics 

2.2 Attitudes 



t. ACROSS LEVEL POLICIES AND MANDATES 



1.0 State graduation requirements 

2.0 State graduation tests 

3.0 Teacher certification requirements 

4.0 Competency testing programs 

5.0 Incentives for school Improvement 



C. ECONOMIC CONDITIONS 



1.0 Growth 1n personal Income 

2.0 Consumer price Index 

3.0 Community employment patterns 

4.0 Employment rates 

5.0 Community ability to pay 
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CONTEXT VARIABLES 
DEMOGRAPHIC 
1.0 Students 

1.1 Characteristics 

a. Raclal/tthnlc composition 

■ : b. Percent of student* speaking a language 
other than Engl isl inane 

c. Percent of students from families receiving 
AFOC * 

d. Parents' educational attainment 

e. Parents' occupation 

f. Percent economically deprived 

1.2 Enrollment 

*• £* rc * n * °f students enrolled continuously 
w throughout the year * 

b. Percent of pupils Mho are non-minority— a 
surrogate for cultural difference 

c. Percent of pupils Mho pay full lunch price— 
a surrogate for socioeconomic level 

d. Percent of pupils Mho are not deficient in 
English-* surrogate for bilingual complexities 

;. e. Ratio of student enrollment to population 

ty** (NCES) (ELC) 

1.3 Attendance 

*• Average percent of pupils present each day 
^(•S*** the district and 
school levels) (elc) 

b. Truancy rates 

- c. Percentage of pupils in average daily 
v. attendance 
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pWSS LEVEL POLICIES MO MANDATES 
||:0 State Graduation Requirements 
|2.0 SUtc Graduation Tesss 
3.0 Teacher Certification Requirements 
4.0 Competency Testing Program 
fS.O Incentives for School Improvement 
.ECONOMIC CONDITIONS 
|t.O Growth in Personal Income 
;2*0 Consumer Price Index 
|3.0 OomuMty Enploynent Patterns 
^4*0 Employment rates 
3.0 Community Ability to Pay 
5.1 Per capita Income 
5*2 Assessed value per chiW 
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Policy and Research Issues in Using Data 
to Promote School Reforms 



Michael Kir«t 
Stanford University 



The negative economic factors such as you experience in the Northwest 
with various kinds of natural resources, some losses of public 
confidence, competition for education from child care funds, and so 
forth — these kinds of problems are widespread. The situation with 
education funding is that it lagged behind inflation between 1980 and 
1983, and then it turned around from 1983 to 1985 and is now going up 
much faster than inflation nationally and in most states. What has 
turned the funding situation around is the educational standards 
movement, the educational excellence movement, and a feeling that these 
higher standards will improve education in a major way. And supporters 
of increased education funding naturally want to see results. I attend 
the Southern Governors Conference every year. The action down there is 
unbelieveable in terms of raising taxes to support education. And the 
governors are now saying to the education community, we need some results 
from you people to show that we're going romewhere. 

To respond to such needs, the question is, how do we get some 
sophisticated results, and where are we in the state data system in our 
attempts to get from here to there? I'll describe what we're doing in 
California, not as a particular model, but just to show you what's going 
on. First of all we need to understand that the nature of these state 
reforms is quite different from what we've seen in the past. Most of 
what we got in the past were programs. The big money was spent on 
programs for target groups. There were programs for the disadvantaged, 
the handicapped, the bilingual, sometimes preschool and so on. In 
contrast, what we have in this reform is a sort of omnibus bill, a welter 
of activities that aren't interconnected as a program, He have more 
testing, as we heard this morning. We have higher graduation standards, 
we have longer days. We have teacher career ladders, teacher incentive 
payments and we have various requirements for teacher evaluation. 
Sometimes we have an old-fashioned program, such as a new preschool 
program, or we have a whole set of activities that are not like the 
Title 1 program. 

Generally, funding of these new reforms has not involved specific 
amounts of money tied to specific interventions. Instead, the university 
raises its standards for entrance, and then there's an increase in 
graduation requirements at the state anu local level. Nobody has costed 
that out; rather there's just a flow of money out of the general fund 
from the state level to support some of these activities. That is, the 
amount of support is not based on figuring out what it costs to raise 
graduation requirements, or what it costs to put in a particular kind of 
teacher evaluation system. It's much vaguer than that. So, we don't 
have programs per se and that' s a problem. 





As I looked at the state data systems, I realized that the reason 
Martha couldn't find a lot of the data was that the state data collection 
procedures are selective. They tend to follow current policy rage. What 
have we been doing for the last ten years—prior to the 1983-1985 
reform—had to do with categorical programs. So we collected data about 
categorical programs. Thus, we knew a lot in California about 
handicapped children, but we didn't know about what our teachers were 
teaching out in the field or what their qualifications were. We could 
tell you a great deal about categorical programs, and we could tell you a 
great deal about finance-equity issues — which had also dominated our 
agenda. But we had very little data about upgrading academic standards 
and improving the quality of th^ teaching force. In fact we didn't 
collect much curriculum data at the state level after the early '70 *s; 
and instead we went gung-ho on categoricals. 

In addition, state data collection efforts tended to be 
uncoordinated. One division collected something from a categorical 
program for limited English speaking children, and another division 
collected a few things about the numbers of certified teachers. But no 
one was putting these data together at the state leveJ or working to 
bring these data tr earns together to do policy analysis and establish 
some kind of "accountability program. " There continue to be huge gaps. 
For example, we have almost no data in most states about middle schools. 
We tend to have more data about testing at the elementary and senior 
secondary levels, but the middle school remains a sort of dark continent 
of educational research and information. We also know very little about 
the community college interface with the elementary-secondary sector. 

Further, we knew very little about teachers. Not long ago we thought 
there wasn't any need to collect data about teachers in California. We 
thought we had a surplus of teachers, and there wasn't any particular 
concern about the quality of teachers from the standpoint of state 
policy. So all we could say was how many of are teachers were licensed 
every year— that kind of information. We don't know who's teaching 
where, we don't know about the nature of the teaching force in any depth 
at all, and we find people very frustrated about this lack of meaningful 
information. 

Moreover, there is a lack of local commitment to using the state 
data. Local people seem to feel, "Ah that stuff we send up to the state 
capitol, most of it doesn'* relate to us. It's sort of a pain in the 
neck. We fill out the forms and that's that." There is not a local 
district and school site feeling of "Gee, these state data are really 
useful, and I use them for planning, so I really want to get my data back 
and fill it out accurately." One of the studies we did of our California 
Basic Education Data System involved asking who fills out the data 
gathering form and how much time do they take? We found that the forms 
get filled out in a quick, rather slapdash way, not with the kind of care 
you would have if you were going to base a local decision on it. Then, 
when we published and used these data, as I'll show you later, local 
people were saying, "This is all inaccurate. This isn't our school at 
all." And the state is saying, "Well, you sent it to us," so we're 
beginning to get a little bit more local commitment as the data are 
coming back to the state on a school by school basis. 
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Nearly all of the states that instituted these kinds of reforms just 
loaded the st*te data collection responsibility on top of the existing 
State Department of Education staff. They didn't expand the staffs, nor 
their analytical capacity in terms of dollars. Therefore, most of the 
state governments across the country report that they are overloaded; now 
states are beginning to get some good data in, but there is nobody 
available at the state level to analyze it. 

Let me move now from these general comments about state data systems 
and talk about where we need to go from here. I'll speak first in terras 
of evaluating recent reforms and the impact of these state actions, and 
then give some specific examples. 

Let me talk about some basic orientations one can use in terms of 
data systems. One of them is that the whole education system needs 
uplifting, and therefore what you need ie a data system which looks at 
how the whole system is progressing. That's the view held by California, 
Texas and, to some extent, Connecticut. Another orientation you can take 
is that the school system is pretty good? that what we have are big 
problems in a few local places, so we need to find the "bad guys" at the 
local level and use the state data system for that purpose. That seemed 
to be the main interest in Wisconsin; they wanted to "get" Milwaukee. 
They weren't particularly interested in the overall system, so they 
wanted to find these good guys and bad guys out there. 

A third orientation you can take is that we basically trust the local 
professional educators to do the job, and therefore our approach will be 
to feed them back the state and local comparative data and let LEA' s do 
what they want with it. Prom this orientation one would not engage in a 
lot of state publicity, statewide targets, or comparisons among schools 
and school districts. 



A fourth approach would be to decide not to rely on the educators a 
lot; to roly, instead, on the aroused citizenry and the parents, the 
business groups— the noneducation interests. Prom this perspective, 
state would have an enormous amount of local disclosure and give this 
information to the citizens in a reasonably readable form, an easily 
packaged form that they can understand. Florida has gone that route: 
They have a local school performance report for every school and they 
have a set of standards. These are mailed out to all the parents, 
published in the local newspaper, and Florida gives out awards every year 
for the most informative local reports by groups of districts. 
Woodville, Florida, haf- the best rural local report, and so on. That's 
basically a sort of bottom-up citizen approach to school improvement. 

While I favor mixture of bottom-up and top-down approaches, most of 
the states' approaches have involved trying to assess the effects of 
multiple state reforms, and they're noz quite sure how the different 
reforms interract. For example, California was providing incentives for 
teachers to experiment in their classrooms, while et the same time 
requiring U 3e of a state curriculum. The first group of policies seemed 
likely to attract good people, and the seca.d may repel them, so how 
would these two interact? 



Let me talk now about seven approaches that I f ve been thinking and 
writing about for assessing state reform. The first approach that the 
states are being advised to use is some system of performance 
indicators. I'll come back to that and hand out the California example. 

The second one involves asking the states to find out what the 
financial impact of these reforms has been; how spending has been 
affected. How do the internal local financial allocations meet these 
reforms differ by the type of school district you have? What do all 
these reforms and the state distributions to LEA's do to your state 
equalization plan? For example, when California pays all the money for 
mentor teacher s, we give the same amount to Beverly Hills as we give to 
the poorer school districts. Meanwhile, we have a state equalization 
system. How do these policies fit together? 

Another thing you can do is group programs by similar objectives and 
then compare their costs and effects. California has at least nine 
different approaches, for example to attract better teachers at the 
beginning J >vel. We've raised the minimum salary from $14 , 000 to $19,000 
in two yea.., we have loans, we have scholarships, and we 1 re spending a 
lot of money on those. We've created the mentor teacher approach, we 
have a teacher trainee approach where we bring in people who do not yet 
have the normal education requirements. We can't fund all of these 
programs fully. Then we have the teacher mini-grant program, and so on. 
Which of these are the most cost effective? States should study this. 

The fourth approach involves looking at specific programs. There are 
imbedded in a lot of these state reform packages of the old-fashioned 
type of program that would fit program evaluation. You could look at a 
presunool program, such as many states have, and you could isolate it as 
a program and begin to evaluate it under that technology. 

Fifth would be to study the impact of several different state 
initiatives aimed at the same general objective, without comparing cost 
and effectiveness. One of the papers I ? ve brought along has to do with 
the curricular change in California high schools in the last two years. 
Now I don't know what's causing all of this change, but we're seeing a 
lot of it. In comprehensive high schools mafch enrollments are up over 20 
percent, science enrollments are up over 20 percent, advanced placement 
enrollments are up over 34 percent, home economics enrollments are down 
22 percent, vocational education/ industrial arts is down over 20 percent, 
business education is down also. Then we see big changes within the 
subjects; for example, social studies is up only marginally, but there is 
a big movement towards world studies and away from the newer social 
science electives of psychology, sociology and so on. 

So, a lot is happening. Now, you can call that an input, or a 
process, or whatever you want to call it, but the legislators are very 
interested in that kind of data. They're saying, "well, maybe they do do 
something out there... .maybe this isn't just flushing money down while 
LEA's attend to the same old business." Now, I don't know what's taught 
in those courses— that's our followup study. California had a teacher 
shortage, as far as we knew, before science and math enrol lments went up 
by 22 percent in the last two years, and I'm wondering who in the world 
O is staffing these classes. 
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In California, at PACE, we follow 22 schools in depth—schcol that 
are representative in terms of their distribution and type across the 
state. I'm not sure that every school in the state can collect all these 
data. My view is that we oujht to go in depth in some selective schools, 
such as the 22 we have been following. Moreover, the state policy people 
seem to think that theae are adequate for at least gauging the initial 
impacts of their policies. Another thing that's revealed with that kind 
cf study is that the biggest jumps in curricular change are in the lowest 
socioeconomic status schools. That makes interesting newspaper headlines 
and it's logical when you think about it. Where is Palo Alto High school 
to go in terms of increased academic course offerings? Most of the kids 
are already taking college preparatory courses. But if you go over to a 
black high school in Oakland, you'll see that they are undergoing quite a 1 
cirriculum shift toward more academic courses. 

The sixth orientation involves what I call aggregate and cumulative 
effect studies. We're just now beginning these in California. Because 
these recent reforms have a whole bunch of initiatives in them, and you 
can't isolate £he interaction effects of all of the reforms, what you 
need to do is study the effects of all the categories of reforms on 

individual schools. The way you do that is to use sample schools — 22, 25 j 
of them—and you look at things like the bottom-up enthusiasm, commitment 
to these state reform efforts, whether they even notice it, etc. So you 
look at the system's response in terms of the schools' commitment, 
interest, knowledge, and then follow over time how they change. 

The last kind of study might be a cause- and-ef feet study, where you 
would attempt to sort out the cause and effects of all of the school site 
changes that are coming about. These kinds of studies are, I think, very 
far off. I'm not sure you'll ever be able to determine causes *nd 
effects precisely, because there are so many causes. Instead, I think 
we're going to be looking at changes and impact in a more general way and 
less at cause and effect. 

Let me talk, then, about the attempt in California to create a 
performance indicator system. This handout is what each school gets, and 
then there's statewide handout. (See the attachment to this 
presentation.) This is the brainchild of Superintendent Bill Honig. His 
view was, "How do I keep the interest in reform and increases in state 
jnoney going?" We all know that policy cycles run very fast in public 
education, and in California the fads come and go even more rapidly, so 
he's trying to keep education in the limelight. 

The basic strategy is that many inidicators are better than a few. 
Many new measurement items are process oriented, such as time spent on 
curriculum, and those are interesting, as well as pupil outcome 
variables. You also need to do some kind of comparisons with similar 
schools. California schools, for better or for worse, are grouped into 
five competing leagues. Those leagues are around parent socioeconomic 
t status and social context of the schools, so each school competes both 

statewide and in its league. Now in gathering data, we in California 
cast about in a hurry to figure out what the SEA had that could be put 
into a local site performance report, and found that we had more than we 
thought. 
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What are some of t a problems with this k,\id of data display used by 
California? First, you have a problem of data reliability. The schools 
that didn't collec*- accurate course enrollment data are finding that it's 
coming back to haunt them. Data validity is a second 
problem—determining whether you're really measuring anything important 
here. This format also tends to obscure things. Larry Cuban, another 
professor at Stanford, says state performance indicators are a collection 
of quantities which only serves to feed the American mania for simple 
indicators like runs batted in and miles per gallon. There are all -nese 
aggregates, 4>,id th* assumption i& that more education is better quality. 
All this obscures what is actually going on at the local level. Reform 
will never take hold, he says, until the teachers really want it an<: will 
g^ab ahold of it. 

On tbe one hand, studies of aggregate effects give you a picture that 
things are moving ahead. There has oeen very little risk for educators 
in the early years of the* studies, because we art moving up. I think 
there's a general tightener g of the curriculum across the country, and so 
education can look good on these curricular indicators. These findings 
have been received with quite soom* interest by a lot of state 
policymakers. The California governor has been fully funding the 
education reform program, saying, "They seem to be shaping up out 
there." On the other hand, Cuban charges that these Ktnds ..£ data 
displays give too much cf a surface illusion of change, when there really 
may be no change in the content taught. Other Stanford professors go on 
to argue that the increased science curriculum may be drivel and turning 
the kids off to science. 

There were several things on which State S jperintendent Honig wanted 
data, but couldn't get the data statewide. These were to be in local 
performance reports. Thus, said, in essence, "Okay, this is a 
statewi^ performance report, I agree it is sort of superficial; but now 
we'll have local performance reports to supplement it. And the local 
performance reports were supposed to have such things as quality of the 
instructional program— including what kind of curriculum items are 
actually taught so as to get < ieper into what's behind these state course 
enrollment trends. There were to be arious kinds of school climate 
measures also, such as whether kids feel safe at school, the morale of 
th^ teachers, and so on. And, waiting— how much writing is goirs on? We 
collect information on writing statewide, but we've ha<? trouble measuring 
increases, because of definition problems. 

Homework is a hot topic these days. People are talking ibout how the 
Japanese kids do thr*<* tiites more homework, and our kids are watching 
Miami Vice, anJ Dallas. So what you need to measure is cha es in 
homework ov^r time. California data indicate that kids art Joing very 
little hranework, but that it's been going up recently. Honig decided not 
to report that statewide; instead it shows up mostly as a local 
indi a tor. 
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Ai other iraicator category is measures of commwucy support and 
parent participation, such as how many parents participate in the 
schools. Some school districts have been running surveys on Fablic 
confidence and community views of the schools in t\e local area. And you 
can look at awards and recognition of the local schools? have they gotten 
special awards? Another category would be participation »n<3 
extracurricular activies by pupils. And, lastly you can review various 
kinds of special needs programs. 

Let me sum up by saying I think the educational data bases were 
caught short in most of the states when the recent reforms began. The 
reforms changed the agenda of education from categorical programs, school 
finance-equity, and standard input indicators such as class size, to 
academic standards and teacher quality. Those are the two keystones as I 
nee the reforms— academic standards and teacher quality— and most state 
data systems were not really focusing on these. The reform groups I 
talked to across states are very aware of the comments made in a new book 
by David Cohen and Ellie Parrar, The Shopping Mall High School . For 
example, state policy focus in the junior and senior highs is on the 
middle tracks, the groups that were not having problems in terms of basic 
skills. The theme you hear from reformers is that we're trying to raise 
the whole level of the U.S. education system. To do that you have to 
focus on that middle group— the group that has traditionally been taking 
two academic courses and three or four other courses, then working 
half-time pumping gas, and not making a lot of trouble. You don't hear 
much about this middle group. That's why this attempt in California to 
assess the achievement by quartiles was very, vary crucial. My daughter 
goes to a school that looks very good on the average, but that's because 
they have many high achievers who bring the achievement averages up, 
while the rest of the school docs poorly. These breakdowns by quartile 
achievement represent a way to get at that disparity in school 
performance. California and other states are spending much more time on 
disaggregating the pupil population within schools by race and by 
quartile and by track, so as to look at them in a more sophisticated 
performance way. So far, state dat.: systems have not kept pace with this 
new policy agenda either. 

So, it's an exciting time to be in the data game. Things are 
changing rapidly and in a very interesting fashion. In my view 
California is doing some good things but I rlso I worry about the 
simplistic nature of seme of it. I think we have to respond carefully *o 
the changes that are happening, cautiously avoiding overrimplification. 
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1984-85 HIGH SCHOOL PERFORMANCE REPORT 
TWO PAGE SUMMARY 



This document describes the contents of the two page sugary of the 1984-85 
High School Performance Report, and it should be read completely before 
attempting to interpret the two page sunmary. Included here are: (1) an 
explanation of the two page summary; (2) a sample copy of a two page summary; 
and (3) a copy of California's statewide targets. 

All quality indicators and additional school information which appear in the 
full-length School Performance Report are contained in the sunmary. 
Information on the summary is grouped according to subject. Numbered boxes are 
drawn around these groups to facilitate the following brief explanation. 
Complete explanations are contained in the full reports sent to schools, 
districts and county offices. 

Quality indicators are presented with the change from the prior year, whenever 
prior year information was available. Percentile ranks of quality indicators, 
which compare each school with all other schools in its performance group, are 
shown. Percentile ranks are not provided for data considered to be additional 
school information. 

1. The box numbered 1, in the upper left corner of the first page contains 
identifying information, including county, district and school codes and names, 
and a code assigned by the College Board to schools. This box appears on the 
second page, as well. 

2. The second box, in the upper right corner, displays twelfth grade 
enrollment, the number tested in the twelfth grade by the California Assessment 
Program (CAP), and the school's membership in a performance grouo, ranging from 
A to E, for the last two years. The performance group assignment is based on a 
measure of parent education, with E group schools representing more highly 
educated parents than A group schools. This box appears on the second pace, as 
well. v v 

3. Percents of students are displayed who reported on a CAP survey that they 
took more than three years of math, four years of English, three years of 
science, four years of social sciences, three yesrs of foreign languages or one 
year of fine art. Percents of students taking physics, chemistry, and advanced 
math courses are reported from the California Basic Eoucation Data Systems 
(CBEDS) survey. The percent of enrollments satisfying the University of 
California A-F requirements and the related percentile rank are shown. 

4. California Assessment Program scores for reading and math are contained in 
this box, including percent correct scores and percents of students scoring in 
the top one-fourth statewide (above QJ), in the top one-half (above Q2), in the 
top three-fourths (above Ql). 

56 



44 



5. Scholastic Aptitude Tfst (SAT) score and Advanced Placement Test scores are 
presented in the fifth box. The SAT quality indicators include the verbal and 
mathematics test scores, the percent of students scoring above 450 on the 
verbal test, the percent scoring above 500 on the mathematics test and 
performance on the achievement tests, including the Test of Standard Written 
English, English Composition, Math Level \ M.^th Level 2, and American 
History. Other school information (not quality indicators) displayed here 
includes the percent scoring above oOO or 600 on the verbal test and above 550 
or 600 on the mathematics test, along with the number and percent of seniors 
taking the SAT. The number of students contributing to each of these school 
averages is shown, along with percentile ranks for the quality indicators. 

The number of students scoring three or better on Advanced Placement 
examinations was obtained from the College Board. The percent of seniors with 
these "passing" scores is displayed. 

6. The school attendance rate, computed from CBEDS as t^e percent of students 
present on October 17, 1085 is shown. 

7. Students were classified with regard to English language fluency (limited 
versus fluent or native) and with regard to mobilitv (moved into the district 
in the tenth grade or later). The percents of students in tl.esr- groups and 
their reading and math CAP achievement scores are presented. These data are 
considered to be additional schoo" information and not quality indicators. 

8. The numbers of students attending either California State University a the 
University ^of California and their grade point averages were obtained. A 
"difference" grade point a/erage v/as computed which represents a comparison of 
the students from this pa-ticular school with all California students. These 
data are considered to be additional school informatic and not quality 
indicators. 

9. Counts and percents of students from arious ethnic groups are displayed by 
grade level. 

10. Numbers of students from difterent ethnic groups who are enrolled in 
physics, chemistry or advanced mathematics courses are displayed. Rates are 
computed as follows: 

a) The numerator is the number of students from a given ethnic 
group in a given course multiplied by 100. 

b) The denominator for physics and chemistry is the number of 
students from a given ethnic group in grade 12. 

c) The denominator for advanced math is the number of students 
from a given ethnic group in grades 11 and 12. 

MFtkgp 
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INFORMATION SERVICES 



HIGH SCHOOL PERFORMANCE REPORT SUMMARY. 198(1-85 



OAT I: Or/09/85 



C 
D 

S 

CB 



1984-85 GnAOC 12 INROLLMENT: 310 

1984-85 GRADE 12 TESTEO: 299 

1984-85 PERFORMANCE GROUP: 0 

1983-84 PERFORMANCE GROUP: 0 



COURSE ENROLLMENT, 1984-85 



COURSE 

3+ YRS MATH 
ADVANCED MATH 
4+ YRS ENGLISH 
3+ YRS SCIENCE 
CHEMI STRY 
PHYSICS 

*♦ YRS SOCIAL SCIENCrS 
3+ YRS FOREIGN LANGUAGES 
I 1+ VR FINE ARTS 
PERCENT IN A-f COURSES 





CHG. FROM 


PERCENTILE 


PERCMT 


I983-84 


IN GROUP 


75-7 


7.8 


75 


33-7 


6.1 


66 


90.0 


9-7 


78 


38.7 


-1.5 


76 


W.7 


M 


89 


13.5 


-2.2 


69 


22.9 


8.2 


37 


41.0 


7-7 


98 


5*. 2 


-7.3 


18 


35.9 


NOT APP. 


30 
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PERFORMANCE OF COLLEGE BOUND STUDENTS 



SCHOLASTIC APTITUDE TEST 
SENIORS TESTED 



VERBAL SKILLS 

AVERAGE TES1 SCORE 
TESTS WITH SCORE>*450 
TESTS WITH SC0RC>»500* 
TESTS WITH SC0RE>"600* 

MATHEMATICS SKILLS 
VERACE TEST SCORE 
TESTS WITH SC0RE>-500 
TESTS WITH SC0RE>"350* 
TEST ^ITM SC0RE>«60C* 

AC HIEVEMENT TESTS 

AVERAGE STO. WRITTEN ENGLISH 
AVERAGE ENGLISH COMPOSITION 
AVERAGE MATH LEVEL 1 
AVERAGE MATH LEVEL 2 
AVERAGE AMERICAN HISTORY 

ADVANCED PLACEMENT TEST 

TESTf WITH SCORE OF 3/BETTER 



NO. OF 


1983-8*. 


P'UtNTlLc 


TESTS 


RESULTS 


IN GROUP 


151 


••9.5* 


NO? APP. 


• 51 


k\S 


33 


56 


18. di; 


38 




11.8% 


NOT MIL. 


6 


2.0* 


NOT AVAIL. 


»5» 




50 


C8 


22.3* 


95 


39 


12.8* 


NOT AVAIL. 


2d 


7.9* 


NOT AMI L . 


151 


41.9 


29 


50 


482 


3«. 


49 


5M 


69 


6 


660 


68 


20 


I.59 


18 


16 


5-2* 
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CALIFORNIA ASSESSMENT PROGR'** 1984-85 



SU8JECT 



READING SKILLS 
PERCENT CORRECT 
PERCENT SCORING "JOVE Q3 
PERCENT SCORING ABOVE Q2 
PERCENT SCORING ABOVE Ql 

MATHEMATICS SKILLS 
PERCENT CORRECT 
PERCENT SCORING ABOVE Q3 
PERCENT SCORING A80VE Q2 
PERCENT SCORING A80VE Ql 



PERCENT 



63-3 
26* 

52* 
79* 



70.2 
30* 
52* 
81* 



CHG. FROM 
1983-S4 



0.6 
-2 
-1 

0 



1.1 
2 
0 
2 



PERCENTILE 
IN GROUP 



35 
40 

35 
49 



63 
73 
69 
6) 



6 



ATTENDANCE RATE (PERCENT) 



100.0 



NOT APP. 



88 



LANGUAGE PROFICIENCY AND MOBILITY ON TEST SCOPES, I984-85* 



LANGUAGE PROFICIfcNCY AND MOBILITY 

LANGUAGE, ALL 

LIMITED ENGL ISM 

FLUENT ENGLISH £ ENGLISH ONLY 
MOBILITY (GRADES 10-12) 



PERCENT OF 
STUDENTS 

100.0 
5.0 
95.0 
17.0 



PERCENT CORRECT FOR: 
READING I MATHEMATiCi 



63.3 
38.1 
64.8 
63.4 



70.2 
61.4 
70.8 
73.7 
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COLLEGE PERFORMANCE, 1982-83* 



FRESHMAN mt;FORMANCE 

FRESHMAN GRADE POINT AVERAGE (GPA) : 
AT UNIVERSITY OF CALIFORNIA(UC) 

CALIFORNIA STA1E UNIVERSITY U$U) 



LAT UN 
AT CAI 



NO. OF 
STUOENTS 



26 
30 



AVG. 
GPA 



2.92 
2.69 



DIFF. OF 
GPA 



0.03 
0.1? 



ITEM IS FOR THE PURPOSE OF PROVIDING ADDIYIONAL INFORMATi 

ERLC>S 



ON FOR THE bCHOOL ANO IS NOT USfO AS A QUALITY INDICATOR. 
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INHUMATION SERVICES 



HIGH SCHOOL PERFORMANCE REPORT SUMMARY , 198^-85 




OAT E : 09/09/85 



I98U-85 GRAOE 12 ENROLLMENT: 3 10 

198U-85 GRAOE 12 TESTEO: 299 

198U-85 PERFORMANCE GROUP: 0 

1983-8<» PERFORMANCE GROUP: 0 



ETHNIC 
GROUP 



WHITE 
BLACK 
HISPANtf 
ASlAN/r aiPiNO 
AMERICAN INOIAN 
AIL 5TU0ENTS 
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ETHNIC 
GROUP 



WHITE 

9LACK 

HISPANIC 

ASIAN/FILIPINO 

AMERICAN INDIAN 

ALL STUDENTS 



ETHNIC DISTRIBUTION BY GRAOE LEVU, 198^-85* 

.1 



GRAOE 12 GRAOE 
NUMBER I PERCENT j JIUMSUR | 



PERCtNT 



G°ADE 



»0 

PERCENT 



G r *.DE 
NUMBER I 



9 

PERCENT 



SCHOOL 
NUM&fR I PERCENT 



209 

22 
63 
0 

310 



bf.k 
3.5 
7.1 

21.9 
0.0 
100.0 



229 


71.8 


;*9 


73.9 


322 


7K2 


1.0*9 


71 


■ 3 


8 


2-5 


6 


1.5 


1 1 


2.'i 


36 


2 


.1* 


21 


6.6 


2U 


6.1 


u\ 


9.1 


108 


7 


.3 


b3 


18. '« 


<9 


17-7 


11 


17.0 


273 


)8 


5 


2 


0.6 


3 


0.8 


1 


0.2 


6 


0 




319 


100.0 




10'". 0 


*52 


1O0.0 


U'*72 


100 


0 



ENROLLMENT RATES OF FTHNIC GHUUPS IN SELF C TED CO.fRSES FOR tiRADE 1\ 198^-85* 
PHYS'.Cs I 



NUMBER 



RATE** 



CHEMISTRY 
NUMBER I RATE** 



AOVANCEO MATHEMATICS 
NUMBER I SME^ifL 



23 
2 
0 

17 
0 
1.2 



!1.0 
18.2 
0.0 
25.0 

13^5 



106 
1 

5 

36 
0 

11*8 



50.7 
9-1 

22.7 
52.9 



10 
3 
7 

*3 
0 

123 



16.0 
15.8 
16.3 
33.9 

0.0 

19.6 



ITEM IS FOR THE PURPOSE OF PROVIDING AOOITIONAL INFORMATION FOR THE SCHOOL ANO IS NOT USEO AS *J QUALITY IKOICATOR. 

** 

PHYSICS ANO CHEMISTRY RATES ARE BASED ON PER 100 SENIORS ENROLLED. 
Aft* 

AOVANCEO MATHEMATICS BATE IS BASED ON PER 100 JUNIORS AND SENIORS ENROMEO. 
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BEST COPY AVAlLABLt 



PAGE 2 Of 2 
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Quality Indicator 



- ^ !if I" -*'' 



A. Course Enronmentt 

Mathematics 

3 or more years 

Advanced Mathematics (per 100 juniors 
and seniors enrolled) 

4 or moreytart 

* ocience 

3 or more years . 

Chemistry (par 100 seniors In school) 
Physics (per 100 seniors In school) 

•'!•:'* 

4. History/Social Science 

4 or more years 

6. Foreign Language 
3 or more years 

6. Fine Arts 

1 or more years 

7. Enrollment In a-f courses 
Percentage of school enrollments in a-f 

courses 

Percentage of graduates completing a-f 
requirements 

8. Units required for graduation 

9. State Board Model Grsluatlon Standards 

Pnrcentage of graduates meeting standards 



BE31 COM V6#fc#V INDICATORS AND STATEWIDE TARGETS 
FOR CALIFORNIA HIGH SCHOOLS 



Dnta first collected in 1984 as 

62 



STATEWIDE AVERAGE 



1983*84 



67% 
28 



73% 



33% 

25 

10 



33 % 
22% 
65 % 



1984-86 



74 % 
32 



86 7o 



bo % 

31 

12 



37% 
22 % 
67 % 
38 % 



STATEWIDE TARGETS 



1985-86 



1987-88 



70 % 
32 



75 % 



38 % 

27 

16 



38 % 



25 % 



66 7» 



40% 



-TO BE ANNOUNCED (Spring 1986)- 

-TO Bl ANNOUNCED (Sprinq 1986)- 
-TO BE ANNOUNCED (Spring 1986)- 



73 % 
36 



78 % 



42 % 

32 
21 



42 % 
29 % 
60 % 
45% 
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1989-90 



75 % 
40 



80 % 



50% 

40 

25 



50 % 



32 % 



70 % 



50% 



Quality Indicator 

B. California Assessment Program (CAP) 
Scores 

1. Reading 
Percent correct score 
Percent scoring 

above 0 
above 0 j 
above Oi 

2. Mathematics 

Percent correct score 
Percent scoring 
above Q 
above Q 2 
above Ot 



STATEWIDE AVERAGE 



1983-84 



6?.2 

25 % 
50% 
75 V 



67.4 

25 % 
50 % 
75 % 



1984-85 



62.9 

27 % 
52 % 
77 % 



68.3 

29 % 
48 % 
77 7 0 



STATEWIDE TARGETS* 



1985-86 



62.7 

26% 
52 % 
76 % 



67.9 

26 % 
52 % 
76 % 



• Targeis are compared lo 1983-84 standards. 



1987-88 


1989-90 


ISO 7 
DO. r 


64.7 


?7 % 


28 % 


56 % 


60 % 


78 % 


80 % 


68.9 


69.8 


?7 % 


28% 


56 % 


60% 


78 % 


80% 



Quality Indicator 


STATEWIDE AVERAGE 


STATEWIDE TARGETS", 




1933-84 


1984-85 


1985-86 


1987-88 


1989-90 


C. Dropout/Attendance 

1. Dropout rate (statewide averages and 
targets are attrition rates, grade 9 
to graduation) 


29.3 % 


29.0 % 


28.3 % 


26.0 % 


23.5 % 


2. Attendance rate - percent of students 
In attendance oh a specified day 


»• 


S3. 7 % 


94.0 % 


94.5 % 

l 


95.0 % 



Lid 



QUALITY INDICATORS AND STATEWIDE TARGETS 



NOTE: All data on this page are fo» 1983-84 school year; 1984-85 results are not yet available. 
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" " O ' of tape's for information only, no statewide targets have been established 



:• Quality Indicator 

* . * **•:•*»•'-. 
, pVlf,^ ~, m „•■■ 


STATEWIDE AVERAGE 


• • • < ■ •sTA^wi6fetAiK«w^-; ;; ' : " , -.: / ' r •• : 


1383*84 


1985-86 


1987-88 


1989-90 


D. Poffortnanoa of Colteotf Bound stt fdarita 
1. Scholastic Aptltuda t«tt (SAT) 










Percent seniors taking 


39.0 0/ » 


* 


• 


• 


Verbal 

Average score 
Percent of seniors scoring 
at least 450 


421 
15.5 % 


4<i8 
17.5 % 


436 

21.0 % 


444 

25.0 % 


Mathematics 
Average score 
Percent of seniors scoring 
at least 500 


476 
16.5 % 


481 
17.8 % 


489 
19.1 % 


496 
20.5 % 


2. Achievement Tests 










Test of Standard Written English 
English Composition 
Mathematics Level 1 
Mathematics Level 2 
American History 


42.6 
492 
517 
647 
501 


43.0 
503 
528 
656 


43.5 
514 
539 
665 

517 


44.0 

525 
550 
675 
525 


3. Advanced Placement (AP) 










Number of scores 3 or better 
(per 100 seniors in school) 


1 
1 

1 


12.0 


15.0 


18.0 
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Developing State, District 
and School Objectives Prom School Profiles 

Franc' . Hatanaka 
Hawaii Department of Education 



Topic: Ac^unting to the Devil, Fanny Fairbanks, and the Public 

Good afterncx.i. One of the hardy perennials of American education is the 
pursuit of accountability —what it is, how it can be attained, and what it 
means, or might mean, to boards of education and to school superintendents. 

Even though accountability has beer< around awhile, I'm always overcomed 
with the queasy feeling when talking about it becasue that's a word bedeviled 
by the lack of clear definition. Without a definition, my talk this afternoon 
will La like horns on a steer--a point here, a point there, and a lot of bull 
in between. 

So, let me "start at the bery beginning"; let me start with not one but 
three ' c initions of accountability. The first two are offered in jest, but 
the third definition is the one that I would like L„ amplify. 

The fi v st definition of accountability is, or was, personified by Farley 
Fairbanks who died and went to the nether region. There Farley reported to 
the head devil to account for his doings here on earth. The head devil 
listened intently and when Farley was through wit-\ the accounting of all his 
actions, the head devil told Farley he cl ild have hi** pick of three rooms in 
which to spend eternity. The devil openec the door to the first room, and in 
the room were thousands of nn*a standing on their heads on a solid brick 
floor. Farley told the devi. he did not want to spend eternity doing that 
So the devil opened the door to the second room, and in the room wero 
thousands of men standing on their heads on a wooded floor. Although Farley 
thought that standing on his head on a wooden floor was better \han a brick 
floor, he still wanted to see the third room. The devil opened the third 
door, and in the room were thousands of men standing up to their ankles in a 
cesspool of foul substance, The men wore ta*' ng and laughing whiie drinking 
coffee and munching on sweets. 

"That stuff they're standing in is horrible," Fairbanks thought, "but I 
prefer this room to the otl.er two." So he w?^<<ed into the room and the head 
devil slammed the door shut. A few minutes later, the assistant devil coulc* 
be heard hollering: *Okay, Fairbanks and all you guys, coffee break is over, 
o back to standing on your heads.'" 

There is a second definition, of a sort, of accountability, which our 
friend Farley Fairbanks also personified while he was living. 

Aftei an argument with his wife Fanny, Farley stalks out of the house 
telling her that he would be going for a walk. Instead he hurries to the 
nearest disco, meets a gorgeous lady, and spends the entire evening drinking, 
talking, and dancing with here. Soon it's 2:00 a.m. and c osing time. Beforo 
leaving, he asks the lady for a favor. He asks her to reach into her handbag, 
pull out her cosmetic case, and sprinkle both Ms hands with powder. Though 
puzzled, the lady nonetheless obliges and liberally dusts both his hands with 
powder. He thanks her then rushes home. F*nny is livid when Farley enters 
the liyi^ to account Tor his whereabouts. She ft ft 



screams: "Where've you been? Do you know what time it is? What's the matter 
with you?" Parley pauses then calmly answers: "I went to the disco, met this: 
gorgeous gal, and we drank, talked an<* danced till closing time." Fanny 
retorts: "You're lying!" Parley replies: "No, I'm not lying. That's the 
truth." Fanny, still upset, yells: "Then show me your hands!" After 
carefully examining Farley's powdered hands, Fanny says: 'Disco, dancing, 
drinking, gorgeoub lady—all lies! You went bowling with the boy3 again!" 

So much for the escapades of our roguish friend, Farley, and the 
accounting he had to do to the head devil and to Fanny. 

The accountability that I would like to discuss at some lengtfi is linked 
to a purpose of a different sort that is sweeping across all of public 
education. This purpose, which is at once old and new, affirms the importance 
of w,blic education and of making excellence its hallmark. But unlike ti>ose 
that came before it, this purpose also affirsis the need to demonstrate 
excellence in tangible ways. 

Excellence translates into better performance by our students in reading, 
writing, speaking, analyzing, computing, and problem-solving. It further 
translates into broader and deeper understanding of the physical, social, 
political, and ethical worlds, and of the need to develop cha:acter and to 
behave responsibly, it is, beyond dispute, a search for a higher standard of 
conduct, a higher order of thinking skills, and a higher level of personal 
achievement that our students can carry with them through life. 

But along with achieving excellence, need to judge our successes and 
to report on them. We must do more than simply say that we ne? 2 to maV . 
changes, or that we ar<» working on those changes. Significant improvements 
may take time, out we still need to demonstrate and report on our short-term 
successes. 

To that end, I ha?* put in place this fall INDICATORS OF EXCELLENCE for 
the Hawaii's public school system, which iL hov I've operationally defined 
accountability. 

INDICATORS 0/ EXCELLENCE serves two purposes. The first is to set goals 
for the »,tate, districts, and indx idual schools; and the second is to report 
cn the progress that was made to reach those goals. 

As a concept, education indicators gained much of it;; prominence during 
the accountability rovement of the 1970s ketp citizens informed about 
education's results progress, and expendit ireru At the forefront of this 
movement were governors, state legislators and school board members, each 
imploring educators to back their claims of successes with accurate and timely 
data. But the data that followed were sparse , incomplete and sometimes 
unconvincing. 

Dissatisfaction with education's res;>onses led to a flurry of 
initiatives. 
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One, in January 1984 , the U.S. Secretary of Educatior released his first 
Wall Chart of education statistics comparing th*. 50 states and the 
District of Columbia on a number of education variables. A second 
Wall Chart followed in December 1984. 

Two, the policy committee of the National Assessment of Educational 

Progress announced a new policy under which states may volunteer to 
administer standardized versions of National Assessment of 
Educational Progress tests. 

And three, at their annual meeting in November 1984 , the Council of Chief 
State School Officers adopted a policy calling for accurate and 
timely data on schools and for making appropriate and valid 
comparisons of those data. 

The signs all point to an enormous surge in public interest in receiving 
quality data on education. 

But more than that, interest is growing on those aspects of education 
that need to be measured, on the specific indicators and ways in which they 
are to be reported, and on ways to use those indicators to improve teaching 
and learning in the classroom. 

For the Hawaii public schools, INDICATORS OP EXCELLENCE describes 
significant aspects of our school system, such that inferences can be made 
about its effectiveness. A high level of usefulness of an indicator is 
reached when the school principal can inquire into the effectiveness of 
instructional programs and services. The same high level of usefulness and 
use of an indicator is also reached ct the district ^nd state when the data 
c?n set off different kinds of inquiries into crafting effective schooling 
practices and later evaluating them. 

Our indicators are of two kinds — outcomes and school climate. Outcomes 
are the most informative indicator of the quality of education, and the 
primary outcome measure is atudent achievement, as Treasured by standardized 
tests. But grade promotion rates and grade-point averages are also used to 
describe quality. School climate is the setting in which education takes 
place, and among the factors that make up the climate of a school are 
perceptions of students, staff, school administrators and parents about the 
school . 

In all, we have nine indicators of learning outcomes. I'll focus on a 
few. One indicator is the percent of students who are promoted at the year's 
and. For this school year, we've set state goals of 99.5 percent for grade 2, 
99.6 percent for grade 6, 98.4 percent for grade 8, and 95 percent for 
grade 10. Numerical goals were also set for SY 1990-91. In addition, each 
district and each school have set their own short- and long-term qoals. 

A second indicator of learniny outcome is the College Board SAT scores. 
For this school year, the goal is a combined verbal and math average of 853, 
which is several points higher than what was obtained last year. The goal for 
SY 1990-91 is 893, wh<ch will bring us in line with President Reagan's goal of 
each state matching or exceeding its 1972 College Board SAT average. 
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We are, of course, setting SAT goals only for the public schools. 
Private school seniors, whose averages are often rolled into the stated 
average, were removed from our computations. 

Several things can be said about our College Board SAT scores from last 

year. 

Though Hawaii's SAT score improved this year by a total of eight 
points—six on the verbal section and two on the math section—only a small 
portion of this increase can be attached to the public schools. The public 
school score increased by a mere two points compared to the 18-point increase 
registered by private schools. 

Even so, our 2-point increase, while hardly comforting, did snap a 
20-year slide in SAT scores and possibly marked a watershed in that dismal 
history of tumbling scores. 

The 1985 public school results signal two things, the first is we will 
fall short of this year's numerical goal for 853 for the SAT. Obviously, the 
shart and steady gains that we had hoped for will have to wait awhile. 

The other message pertains to priorities for boosting our SAT scores. 
The ifirst of those priorities is to raise the verbal scores, which trail our 
math scores by rteariy 85 points, of all our test-taking seniors; the second is 
to lift the scores of seniors who as nophomores scored in i-ne average and 
below-average stanines on our statewide achievement test. 

Simply put, unless we see a drastic rise in verbal scores of all our 
seniors, in particular, higher verbal and mat.) scores from low- and average- 
achieving seniors, Hawaii's public school SAT scores will stand still. 

Cne of tee nice features of setting numerical goals is that it fosters 
different ways of looking at old problems and of floating new solutions. The 
problem of low SAT acores has been with us for ? long time, but it was only 
after goals were set to raise those scores by a fixed amount chat those 
initiatives arose. Here are some of our solutions that might be of interest 
to you. 

The first is to have our nigh school counselors analyze SAT scores 
of their past seniors in order to pinpoint courses that are associated 
with high SAT scores. 

The second ir to have our high school counselors investigate the 
relationship of hign SAT scores to participation in literary, forensic, 
and other similar activities that stress organization of ideas, logic, 
vocabulary building, spelling, and correct grammar. 

If such relationships exist, our high schools will steer juniors and 
seniors with college planz inw^ ♦'hose courses and co-curricular 
activities. The College Board just issued a report giving credence to 
the relationship between participation in a few well-chosen co-curricular 
activities and later success in college. 
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Still another action is to make our college-bound juniors and 
seniors SAT-wise: arm all of them with an assortment of test-taking 
skills and experiences so that tney can perform efficiently and with 
confidence on the SAT. 

In that regard, we are putting the finishing touches to the idea of using 
Chapter 1 funds to enroll all of our Chapter 1-eligible juniors and seniors 
with college plans into one or more PSAT- and SAT-prep programs. In addition, 
we are about to launch a statewide effort using state funds to enroll juniors 
and seniors in those prep programs— especially juniors and seniors in high 
schools that have historically performed poorly on the SAT. We would either 
hire our own teachers to conduct those programs during non -school hours or 
work with private firms to conduct those programs for us. 

A longer-term idea that we are working on is to require high school 
English teachers to assign a minimum number of writing assignments, say one 
per week, with the assignments to be corrected and returned to students within 
a specified period of time. Schools would be permitted to hire lay readers to 
correct and grade appropriate parts of the assignments. 

By using the College Board SAT to set academic achievement goals, the 
golas became powerful motivators to do things differently. They set off some 
new and unexpectedly high levels of positive actions to have our school system 
equal and even exceed those goals. 

The notion behind this goal-setting activity is that the nore often our 
high schools are held accountable for the performance of their test-taking 
juniors and seniors, the greater will be their effort to improving the 
scores. According to the research, setting high goals ar* communicating those 
goals to the entire school are elements common to effective schools. 

Here is another example of how achievement golas triggered inquiries that 
otherwise would have remained dormant or would have been remanded to the 
backburners of our system. A goal is to increase the percent of pupiU who 
score in thr average and above-average stanines in reading and in mathematics 
on our statewide achievement test, the Stanford Achievement Test. This means 
that there would be a corresponding decrease in the percent of pupils scoring 
in the below-average stanines on the Stanford. None of our schools missed the 
sign, especially the Chapter 1 schools. For Chapter 1 schools to meet their 
school goals, they need to move students out of the belw average stanines — 
students in the 1st or 23rd percentiles on the publishers norm—into the 
higher stanines. 

T^en the goala ignited a spat of questions — questions that probably 
should 1 ve been asked a very long time ago. Thay asked: How is it that with 
all of the impressive NCE gains that the Chapter 1 schools report each year, 
those gains don't show up on our statewide Stanford Achievement Test? Is it 
becaure the tests that our schools use to measure learning gains and to later 
report on those gains in NCE units are out of kilter with our Stanford? Or is 
it because the amount of NCE gains are trivial, thus they contribute little to 
the gains we have sought on the Stanford? Or the policy question: Is this an 
instance of winning the battles but losing the war? 
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Here's another question connected with our Chapter 1 schools that left me 
stunned with pleasure. The question was: How might our principals be 
assisted to channel their Chapter 1 resources, and the state resources that 
they supplement, to instructional activities that have high payoffs? Now, 
that's a key policy question. It is also a key question in evaluating the 
effectiveness of Chapter 1 funds. Those questions, brought together and 
answered together, provide the basis to allocate resources in a way that is 
totally new but exciting to me. With ^ 20-year supply of project 
effectiveness and cost analyses data for Chapter w, my instinct tells me that 
a way to retrieve cost-effectiveness date on a modest scale is within our 
reach. My instinct also tells roe that the search for cause-and-effect data in 
Chapter 1 is not elusive as it appears — it shouldn't be for projects that have 
a plentiful supply of data. 

We are also using our statewide achievement test to set numerical goals 
for the eleme*.<:ary, intermediate, and high schools. Moreover, intermediate 
and high schools have goals calling for increases in the overall grade point 
averages of their students, and high schools have goals that are referenced to 
our Hawaii State Test of Essential Competencies — our so-called graduation 
test. Other indicators around which goals are set are participation rates in 
co-curr icular activitj.es, and adults in our community schools completing their 
programs or receiving their high school diplomas by way of the GED test. 

An area that we have explored at some length and have added to our 
INDICATORS OP EXCELLENCE is the assessment of school climate. Starting la»t 
year, our schools have assessed their 3chool climate using scales that the 
department developed and tested. School climate is the personality or feeling 
that comes out of people relating to each other such that morale, respect, 
ciring, coheriveness and the opportunity to make input all improve. 

School j.imate goals, which are expressed numerically, are our attempt to 
systematically look at and improve teaching, learning, as well as attain a 
satisfying school environment. 

The basis for implanting school climate into our INDICATORS OP EXCELLENCE 
was the research on effective schools. The research pointed to mutual respect 
and trust among students, staff, administrators and parents as one element of 
an effective schools. Other elements were sharing responsibilities, issuing 
clear and consistent discipline policies, providing for parent-initiated 
projects to improve the school, and maintaining adequate facilities and a 
well-kept school plant. 

School climave assessments involve elementary and secondary students, 
teachers, clerical and other support staff, counselors and other professional 
support staff, school principals and vice principals, and parents. In all, 
there are six major groups from whom scores are obtained and later reported 
,ith the help of the state*s main frame computer and a micro computer scoring 
system that the department operates. 

The assessments made last year of nearly three-fourths of our schools 
showed expected and unexpected results. They showed that the farther a group 
is removed from the administration of the school, the less positive are the 
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perceptions of the school's climate. That is, school administrators view 
school climate positively, teachers less positively than administrators, and 
students even less than teachers. But the unexpected results were that 
secondary students and, to a lesser extent, teachers and parents felt the 
opportunity to contribute their ideas to improving the schools was 
conspicuously missing. This pattern was obtained from nearly all of our 
secondary schools. 

That perception, of course, is counterproductive to student self-esteem, 
and it deprives the school administrator of new and fresh ideas to run an 
effective school. 

Prom that experience, I am beginning to feel that the perceptions of 
secondary students are not only harsher than those of the staff and of 
parents, but also their perceptions are hardest to change. It seems to take 
more time, resources, and explanations to turn things around for students 
mainly because students, more than adults, show a striking ambivalence to 
change • 

Measuring perceptions of people and using those perceptions to set goals 
in order to change behaviors is a new venture for us. Perceptions, after all, 
are highly sensitive to the value systems of people and of institutions and to 
events over which our schools have little or no direct control. Indeed, 
questions are asked about the reliability of our scales and the worth of 'sing 
the scores to set goals and to do re-assessments to check on progress. 

Though the problems just mentioned are found in tests of achievement, 
they are compounded in school climate assessment scales. But it is my belief 
that those problems have been controlled sufficiently, and our scales can and 
should be used to set schools 1 goals and later report on progress. 

That's another nice feature of our accountability system. It does make 
us take risks that we would not ordinarily take. 

Enough said for now on the foundation and the main features of INDICATORS 
OF EXCELLENCE, our accountability system. 

Let me show you next the dark side of the system. 

Before I do that, there's a story about the dark side of Lent that I'd 
like to share with you. One evening, a father told his ten-year-old son that 
he should give up something for Lent — something that would be a real 
sacrifice, such as c^ndy. the boy hesitated for a moment and then asked his 
father whe he gave up. 

"I gave up liquor,*' the father replied. "But, dad, I saw you drinking 
something before dinner, 19 said the boy. 

"Yes,* said the father. "That was sherry — I gave up hard liquor. * •Well 
dad," shot back the boy, "I think I f ll give up hard cendy." 
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There are principals in our 225 schools who view INDICATORS as a bane to 
good education. Their objections center on, first, the quality of the data 
that are in our educational data base— actually the quality of the data that 
they, the principals, send to the state office; second, the school-to-school 
comparisons that the board of education, the state legislature, and the 
general public will make from the year-end results; and third, the difficulty 
in setting numerical goals. 

Consider the first objection, that of the quality of t*ie data base from 
which numerical goals are to be set and the current year's data are to be 
added. There are variations from school to school in reporting of data: some 
schools are more conscientious than other schools in recording, say, pupil 
absences and suspensions and then reporting them to the state office. The 
objection among some principals is that those vagaries in reporting data will 
"harm* the conscientious principal and e the same time distort district and 
state averages. 



My simple and straight forward advice to those principals is to worry 
less about what other principals are doing or not doing and to worry more 
about cleaning up their own data reporting systems so that their goals and 
their school results are solid. The main thrust of INDICATORS is to have our 
schools measure up to the goals that they have set for themselves; it is not 
to foster comparisons between schools or between districts. 

As for the second objection, that of school-to-school compar isions, those 
comoarisions a; a apt to happen based on what we've learned from the USDOE Wall 
Chart*. Frankly, I don't know how to totally prevent the public from making 
unwarranted comparisons, but there are ways to steer the public away from 
making them by publieizinb the kinds of comparisons thit should be made. In 
that regard, I don't hold much hope for fostering relevant and valid 
school-to-school comparisons using a whole lot of demographic data. Scanning 
those background data, such as teacher salaries and socio-economic indexes, 
will not yield the insights that are needed to make fair comparisons. That's 
the same problem I f ve encountered with the wall Chart. "Population 
characteristics", such as per capita income, median years of education, and 
poverty rates, or "resource inputs*, such as average teacher salary and per 
pupil expenditures, don't mean much for making accurate and appropriate 
comparisons if we continue to squint here and squint there at the data, which 
is what we're doing. 

But more important still, even the demographic data themselves are hard 
to interpret. In the Wall Chart, for example, Hawaii ranks second nationally 
in the enrollment of minority students— Hawaii has 75.2% minority students, 
the nation 26.7%. To interpret that ranking, it seems another set of 
background data is required, namely the racial makeup of the stated 
population that the Wall Chart does not provide. Another hard-to interpret 
ranking in the Wall Chart is our national ranking on average teacher salary. 
We rank fifth in the nation with $24,319, but that ranking of itself is not 
enough to mak a judgment of the monetary worth of our teachers. Missing 
conspicuously is the backgroun data to that background data: Hawaii's cost of 
living is roughly 20% higher than tho national average. 
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But that's less of a quandary than the one I read about recently. A man 
dying of thirst in a desert stumbles on to a gas station. He begs for water 
and is promptly refused. Instead the station owner offers to sell him a dozen 
neckties for a dollar , which the man dying of thirst spurns. He continues 
walking till he comes *o a mansion with gorgeous waterfalls and several large 
swimming pools. He rings the doorbell and asks the servant permission for a 
drink of water from the many water fountains. But he is promptly turned down 
because he doesn't have a tie to enter the nans ion ground*. 

Golfer Lee Tievino lamented on his quandary on how his fame is fleeting. 
This is how he described it. "A woman had me autograph a $5 bill and told me 
she would treasure it forever. A half-hour later , I bought some drinks with a 
$20 bill and when the change came back, in it was the S5 I had autographed. " 

The third objection from some of our principals, that of difficulty in 
setting numerical goals for achievement, behavior, attendance, and school 
climate, irks me the most. It irks me because numerical goals are being set 
daily for a host of educational activities in our school system. There's one 
for the football season, another for the school fund-raiser to send the band 
to Pasadena to perform in the Rose BijwI parade, and still another to sign up 
students for a campus beautificat on contest among different campus clubs. At 
the root of ther fears is, I think, *t natural aversion for setting goals that 
may later hurt them if the goals are not met. unmet goals in academic 
achievement, student behavior, and school attendance are apt to be more 
threatening than unmet goals in say, football. 

After 30 years, I've learned to spot the danger signals and avoid being 
taken in. So to assuage their fears, I tell them to set their goals one unit 
better than what they obtained the year before. If th* daily attendance rate 
vas 90% last year, they would set this year's goal at :*!%. If suspensions for 
assaults on campus numbered a hundred last year, this year's goal night be to 
reduce it by one percent. Actually, they can set numerical goals if they use 
a variety of technique* that are set forth in the administrator's manual on 
INDICATORS OP EXCELLENT. Those techniques are referenced to historical as 
well as current data on school productivity that the state office annually 
prepares then sends to every school in the state. 

There are at least two other states that have launched goal-setting 
projects similar to our INDICATORS OF QUALITY. 

California has quality indicators for high schools, intermediate schools, 
and elementary schools. 

For high school?*, there are five categories of indicators: 

3.. Enrollment in selected academic courses. 

2. sAchievement scozes* 

3. College entrance exam scores and freshmen year grades in the 
California college system. 

4. Dropout and attendance tates. 
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5. Student participation in extracurricular activities and the 
amount of homework and writing assignments completed. 

Elementary and intermediate have different kinds of indicators. They 
include number of instructional minutes per day, achievement scores in reading 
and mathematics, amount of writing and homework completed, attendance, and 
enrollment in algebra uid science. 

An interesting feature of California's approach is that schools are 
grouped into five different socio-economic categories for makirg and then 
reporting comparisons among the schools. 

Kentucky draws on three types of indicators—output, input, and process. 

Output indicators include vocational education placement rates, college 
entrants, dropouts, retention rates, and standardized testing results from 
statewide tests given in grades 3, 5, 7, and 10. 

Input indicators include assessed valuation per pupil, local financial 
support, total enrollment, and the percent of econonically deprived and 
minority students. 

Process indicators include salaries and rank of teachers, instructional 
cost per pupil, profess ional/nonprof ess ional staff ratio, central 
office/school professional staff ratio, and minority staff ratio. 

The interesting feature of Kentucky's approach is a statistical model to 
explain variations in outputs by factoring in various input and process 
var iables • 

To sum up, accountability is not only compatible with promoting 
excellence in our schools, it is essential to the survival of our public 
schools. Accountability in public education inevitably comes doi/n to a 
willingness to set goals, assess achievements, report on progress, and 
allocate scare resources on the most rational way of which we are capable. 
That's the thrust of our INDICATORS OP EXCELLENCE. One of the interesting 
paradoxes of INDICATORS was that in making some of our present ways of doing 
things obsolete, we revitalized still older methods. Reinventing the wheel is 
a phrase applied scornfully to rediscovering the obvious-- and there is nuch 
cogency to that belief. But when the obvious is half forgotten, it if. well 
worth reinventing even though it's sometimes like trying to sew a button on a 
custard pie. 
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State Activ**iee In Developing School Profiles, 
Data Bases and Indicators 

Harvey Croromet 
Alaska Department of Education 



I'd like to talk about two elements of what we have been "oing in the 
Deptartment of Education in Alaska in terms of our data needs. First, 
I'll talk about what we are doing to get the information necessary for 
planning, for federal reports, and for other things that we have to do, 
and then give you a brief overview of a data framework for school 
profiling that we are doing. 

A year ago the Commissioner asked me if I would chair a group of 
people to survey die Department to see what data we actually did have, 
where it was coming from, who needs it, and other things you have to do 
to set up a data based management system. I was very lucky in getting a 
group of five people from different sections of the Department. We 
dec idea we would do a survey of the program managers and uiers of the 
data in the Department to see what dats they needed, where they got the 
data, where the data came from, what uses they made out of these 
data — and then try to come up with a plan for the organization. In the 
process of doing this we found, as everyone else knows, that the same 
data were collected by many different sections of the Department, but 
without interaction among these sections. We realized we had been 
forcing the school districts to present the same information sever a* 
different times. So one of our key elements was to set up our data base 
so that we could cross-reference it to any application and only have to 
collect the data once. If information were collected under the migrant 
program, for exasople, then it didn't have to reappear under one of the 
special education programs, and so forth. 

We also wanted to come up with a software package that would do what 
we wanted it to do. We looked at many different systems. We have the 
WANG VS300 system in our Department, which is a pretty powerful, 
high-capacity computer, and we found that WANG had just released a new 
data base management system called PACE. I went to take a look at it 
where it was in operation, then we brought in some people from WANG, 
They g&ve us some training, and we decided that the system would do what 
we wanted it to do. First of all, it was quite easy for a program 
manager to be trained to be able to manipulate the data in the base once 
it io put up. If they had special applications that they wanted to write 
themselves, they could do it; they didn't have to be programmers because 
it was a system that generated the program code for you after you had set 
up the perimeters. So we brought the system in during the summer. It is 
now in operation, we have about six good applications on it already, and 
I think it is going to be a real help. 
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We started by going around to the different sections. I will 
describe what we did with one section to explain the procedure that we 
used, we went to our Chapter 1 coordinator and asked "what are the 
sources of the data that you need?" The coordinator showed us the 
application that goes out to the schools, the evaluation instrument, the 
achievement forms that they use, the state perforiiiance report, the 
monitoring document that we use, etc. Using all of these instruments, we 
made a data dictionary and defined each of those data elements—where 
they came from, how they were selected. We put this all together, took 
it back and went over it *rith the coordinator, *ho then pointed out any 
inaccurate items. 

After about three of these go-arounds, the coordinator said "yes," I 
think you have a pretty good handle with what we need and what reports 
will have to come out of it, and so forth. 

During this process we designed our series of data bases: One is 
basically a student data base that we have t<- lave, another is concerned 
with the financial side of it, the third ge into the evaluation, and so 
forth. By tying all of these together witu key elements, we are now able 
to go into it and ask questions without any trouble; we can also generate 
our federal forms out of it just by asking the right questions. It has 
an excellent query section on the program. We are in the process now of 
doing our statewide audit form that each district has to submit to us, 
and we are in the process of putting that one into the system. That will 
be our second application. The third application will be Chapter 2. 
Special education is a little farther down the line, but that vill be 
coming up sometime next year. 

We think the sys' m is going to work for us quite well. Of course, 
it requires a lot of time and effort; nobody works on it full-time. We 
ell have two or three other jobs, so that we sandwich it in between 
thsse. Since the Commissioner of Education has a real interest in the 
system, we find that when we need a week or two weeks of concentrated 
work to do a certain part of it, he makes sure we get relieved from our 
other duties so we can work on it. So, the project Is caning along. We 
are about three months behind our timeline, which I don't think is too 
bad. I am looking forward to having at least three applications up and 
running by the end of the school year. 

The second aspect of our current work is a framework for school 
profiling. One of the things that we really wanted to do was to be able 
to review our information about a given school before we ever went out to 
do our monitoring. However, in attempting to do this, we found that we 
J ad pieces of it all over the Department, pieces which were never brought 
together. So we contracted with Interest and they developed a framework 
for school profiling for us. They have put it on the Apple computer, so 
*hat we are now able to nave the schools generate a lot of the data 
coming directly back from them. Every school in Alaska has a lot of 
Apple computers, so we used it because it is so common; schools can just 
send us a disc. We are consolidating the information and are looking 
forward to putting the profiling sy3tem into operation sometime during 
the next school year. 
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Alaska has only 53 school districts, which is net a great number* We 
have just over 100,000 students in the entire state, so we are talking 
about a relatively small data base. The only problem is that our school 
system is spread ovt ov*t so many thousands of square miles that it is 
hard to transmit data* We do have a satelite system, so that now we have 
telephone communications to all of the villages, and are beginning to 
transmit a little bit of data that way* We have the electronic mail 
system that is online, so we can finally transmit electronic mail to any 
school district in the state* Basically though, we are still usim* the 
mail, sending things in, and doing the keying of it at this time* We are 
discussing with some of the bigger school districts, such as Juneau, 
Anchorage, Fairbanks and Renal and the Mat Su Valley, the possibility of 
transferring data magnetic tape* Most of those districts have IBM 
systems, and we can read the tapes with our WANG if we set them up in the 
right protocol* So, we are in the process of doing some of the 
electronic transfer* 

That is where we are* Whers we want to go is to develop a system 
that would enable us to answer the ad hoc questions that come in from 
legislatures, the governor, and the school districts about what is 
happening in education in the state* Right now, it is almost impossible 
to do it* If we do get a request, we have to go in and do the handcount, 
going through 100 different files to find it* Since that ties up people 
and lots of time, we hope wo can get around it* 

One way that this meeting and meetings like this are going to be 
valuable to us is to give us all a sense of what other people are doing 
with data collection and management* I wish that we could all share our 
data elements and definitions, so that we could really look at comparable 
types of things across the states* At the present time, we don't even 
have it well enough defined in the states to be sure that the count from 
one school district is *he same as it is for another school district, but 
we are getting there* 



so 

63 




State Activities in Developing School Profiles, 
Data Bases and Indicators 



Ichiro Fukumoto 
Hawaii Department of Education 



I have set two goals for myself this afternoon. The first is to talk 
about our data profile, which is now being passed out (see the attachment 
to this presentation). The second goal is to stop talking before you 
stop listening, so I have set my clock for 20 minutes and it won : t go 
beyond that, I promise. 

Our data base is designed to serve a mix of intended and unintended 
purposes. Let me talk first about the intended purposes of that profile 
that was just passed out to you. The first, I think, is to serve, for 
the school principal, some of the reporting requirements that come with 
school accreditation. In fact, all our our high schools and nearly all 
of our intermediate schools are now undergoing accreditation through the 
Western Association of Schools and Colleges. One of the requirements is 
to provide historical data on the school to assist the review team to do 
its analysis and to come up with program improvement recommendations for 
the school. So the data base has been used by principals who need to 
call up information to complete the report that is required by the 
accreditation review team. 

The second purpose is to build up an awareness within the community 
about the school and about schooling in general. School principals will 
use that dat~ base to talk to their own constituencies about what the 
school can offer and what it has offered in the past. One particular 
target group would be parents new to the school or community and students 
that are new to the school. It is a kind of orientation tour. Rather 
than to go through reams and reams of documents, what school principals 
do generally is to pull out whatever information they need from the data 
profile, and then project information onto transparencies for viewing by 
a fairly wide audience. 

The third and probably the most important reason to have the data 
base is to help principals make decisions about the school — program 
improvement, curriculum improvement decisions., decisions about how to 
allocate resources. What they do here is to look at past trends — trends 
in achievement, attendance, behavior of kids, school climai e. Pulling 
all of these data together, they can make informed decisions — to fund 
certain programs or not fund certain programs or projects, cut back on 
others, expand on others, or even terminate some of the programs and 
services. 

As for the unintended uses, one is the goal setting activity that the 
superintendent talked about over lunch today. Principals would look for 
a trend or historical jpattern of achievement, behavior and so on, and 
armed with that information, they would project their short- and 
long-term goals for the school. So while that may be of a fairly good 
purpose here, some of the principals feel that is just not how data bases 
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should be used. They question, as the superintendent pointed ^ut, the 
quality of the data in that school data b*se. But again, you have to 
remember wh»t we have done at the state level is simply to collect 
aggregate information that, for the most part was reported to us by the 
schools. What we have done is to reorganize the data and report that 
information back to the schools. So we don't really give too n>uch 
credence to those kinds of concerns or comments. 

Ohe other unintended purpcse is a kind of brokerage purpose that has 
descended upon people in the district and state offices. This function 
is fairly new to the state office, and it goes something like this: we 
have parents who are new to Hawaii and, like all new parents, they shop 
around for schools. They call us and say, "Tell us what the achievement 
pattern has been for school A and compare it with achievement patterns 
for schools B, C, and D, because those are the four schools that I have 
i.i mind for my youngster to attend.* Now, I would say that is a fairly 
new function, because we have never done this in the institutional 
sense. We have generally referred those parents to the principal of a 
home school or the school that the youngster probably would be attending 
based upon the place or community that that parent would like to settle 
in. Baaed upon the par nts 1 interview with the principal, they wc^ld 
make the decision about whether to snnd the youngster to the school or to 
move into community h versus community B, C, or D. Principals with 
schools that have had a fairly long history of low test scores, poor 
attendance, and a high incidence of all kinds of behavior problems resist 
this whcle effort. They feel that this is not the way to use the cata 
profile. They would prefer, instead, to have those parents come in to 
talk to them and to tell the parents what is, beyond the data base, what 
the numbers don't show. And I nave some ambivalent feelings about this. 
I think the principals have a good point there. On tht other hand, .^or 
us to not Z .zni i the data parents want is a serious problem. If you 
don't furnifth them tte information, one consequence is that those 
youngster b wind up in private schools. They don't come to public 
schools, w*. have something like 18 p3rceni of ^nool-age youngsters 
attending private schools in Hawaii. I think this is one of the highest 
percentages in the nation. 

I will go through the data base itself in the next ten minutes. On 
the table of contents you find what would be contained in a typical 
school data base. On the left column it starts with student achievement 
and goes all the way down to school crime data. These are the four kinds 
of quality indicators we have for the school: data on achievement, 
attendance, behavior, and school climate - On the back of the same sheet, 
you have data on the school's context — student demographics, teacher 
demographics, and some d?ta ou the U.S. cersus by school attendance 
areas. I will touch on this for a few minutes later on. 

v . pages A-l and A- 2 are graphs displaying scores rrom the Stanford 
Achievement Tests administered statewide from grades 2, 4, 6, 8 and 10, 
in reading and math. In addition, we have a profile for reading and math 
by ethnicity (A-3). We show the results on the Stanford Achievement Test 
by each of the 12 ethnic categories tht>t we have in ^ur student data 
base. We have a profile for competency- based measutes there for the 
third grade and for academic efficiency, as it is exhibited by a high 
retention or promotion rate. 



A-4— students x.f limited proficiency. We have an indicator there. 
It is called ar «*xit rate. Abo^t 8 percent of all youngsters are in that 
program. We are tinding that they are getting fairly comfortable rtaving 
in that program. Th*y are liKe long-term tenants, I suppose, and our 
goal is to get them out of the program, we thought a way to do that 
would be to set some numerical goals to step uo the percentage of kids 
who exit the program each year. Grace point ->/erages are provided for 
intermediate and high schools? lower than 2.0 is equivalent to lower than 
a D average (A-4) , J.O is B average or higher (A-5) . Seniors graduating 
with 20 ih-6) credit * would be the minimum number of credits for 
graduation from a four-year high school. We are finding that about 80 
percent of our kids do exceed that 20-credit requirement. 

And, of cheese, we have the opposite end of the continuum. We have 
seniors not graduating (A-7) for one of two reasons: One— failing to 
pas? all HSTEC, the acronym for Hawaii State Test of Essential 
Competencies, which is like w sniminum competency test or a graduation 
test. Also, we have a failure rate for seniors by the number of credits 
and co; .ses that they have failed to meet or to take. 

On attendance, Category B, we have some data on Tie movement or the 
mobility of youngsters. District Exceptions (B-2) would be exceptions 
for which permission is granted by the school system to allow young, _rs 
living in school district A to attend a school in school district B. We 
are saying that they can legally cross these schoci attendance lines. 
This has to be done through a formal process, and the usual reason ij to 
take a course that is not offered in school A,- to move over intc school 
district B to take i course that is offered there instead. Or another 
reason is chilCcare. If I worfc \n town and live on the windward side of 
the island, I may need to get i* child enrolled in the school in town. 

The average daily attendance or absence rate (B-3) , that is a 
conventional indicator which tells you the percent of Mas that was 
absent in any given day. 

Page C-l concerns suspensions, we are trying to reduce the incidence 
of suspensions for crisis offenses and regular offenses. Crisis offenses 
are the ones that are so severe that we can, without consent from or 
consultation with paretics, without even notifying parents, suspend the 
youngsters — take them out of the school for their own safety or for the 
safety of others. The regular suspension, of course, would occur for a 
whole assortment of school-related offenses. The offenses A, B, and C 
(page C-2) , this is something created by tUe Board of Education. It is a 
table of offenses and remedies, I suppose. Offense A weald be the most 
serious kinds of offenses — felonies in the law books* Examples would be 
assault or property damage, offenses which c&U for nugatory police 
action. Some form of police notification has to follow such offenses. 
Offense B refers to in-school offenses such as disorderly conduct or 
theft—even thor<* t>;o would require police reporting. Offense C refers 
to all kinds of move minor offenses that kidc commit in school— smoking, 
swearing, cutting class, being insubordinate, and so on. That would 
represent the majority of offenses about which that schools would have tc 
report anc 1 fc ccountable. 
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Category d is school climate. r,s the superintendent pointed out, we 
have gone into this area for :he first time during last vet.'. We ar» 
having schools report on factors connected with the overal- clinuce of 
the school, m addition to climate, we are talking a Uu<: the qua ity of 
the program, the processes and, of course, the adequacy of materials in 
the school (see D-l) . That is done for ill of the schools, and pupils, 
staff, parents, school administrators ail get into the act. 

Category E give some information shout student demographics. This is 
the contexual data that we send to the schools on a routine basis. We 
prov.Ie ethnicity information, as does California. We also provide 
information on the number and percent of kids in the school whose parents 
work or live on federal installations, the percent of youngsters on 
public assistance, the percent of kids vho participate in free snd 
J^i?^ Programs, the special education snd the students of limited 

English proficiency, and pupils who are now in those two programs. 

Wi also provide information on teacher demographics— eth, icity, 
expedience, age and absences. I think principals find that information 
useful, because it signals certain kinds of moves and administrative 
decisions that need to be made about teaching staffs, such rtS these 
involvii i retirement. Knowing which teachers would be retiring, when and 
what their subject areas are, enables the principals to make long-term 
plans to replace there teachers. 

Refer to Catego.y G, U.S. Census Data. We sent a fellow up to 
Washington, D.C. , for four days to the Bureau of Cansus and one to New 
Jersey to the gambling casinos, to wo.rk out a system wherein we could 
match oui sahool attendance areas wiv-.h ';heu.S. Census track data. We 
are trying to define school attendance areas ant] align those with what 
s»t90t be shown on the U.S. Census maps. Based on that kind of matching, 
the U.S. Census Bjreau was able to fu ish us data about our school 
attendance areas, *nd we have several Kinds of data. For example, we 
have data on the high school graduates among persons 20 years of age or 
older, the percent or number of four-year college graduates in that 
attendance area, the unemployment rate, the average family size, and 
median household income. Probably most important cf all, we have data on 
youngsters from single-parent hoses, which has implications for before- 
and after-school care programs, as well as calling for counselit j and 
guidance programs that would not be found in schools and districts with a 
high percentage of intact, two-parent homes. 

The kinds of data I have been discussing are furnished on a routine 
b«sis annually. It is mandatory that all schools receive this 
il2 C U k ^' . With respect to "medial programs, page A l. a shows that 
1980-81 and the pattern for 1984-85 has not changed for the above-average 
categories of scores, it has, however, changed somewhat for the average 
band of scores. I will skip tb* mzth achieveme.it of our youngsters and 
move on to the next display, Khich is stanine Profile by Ethnicity (page 
A.l.c). There are 14 different ethnic categories on that profile. If 
you work your way across tb-t page free left to right, you find that, for 
reading, there is one *. erican Indian from that school whose score we 
could not determine. Probably the computer rejected that set of answer 
sheets or something happened there— there was no response. That 50 means 
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State Activities in Developing School Profiles, 
Data Bases and Indicators 



Wayne Neuburger 
Oregon Department of Education 



There are two aspects of the Oregon experience that I would like to 
talk about — two things we are trying to accomplish in terms of looking at 
indicators in the state. The first is to look at what we are calling 
goals for 1990, and the second is to look at school profiles. 

In the goals for 1990 , we are trying to look backwards at well as 
forwards. Both of these efforts have come out of the educational reform 
effort in the state which is referred to as the Oregon Action Plan for 
Excellence or OAPE if you abbreviate it. The reform was initiated two 
year 8 ago, by our State Superintendent, and he identified eight areas in 
which he wanted us to take action. *rior to formalizing the OAPE, he 
assigned a series of task forces to look at a number of areas and make 
recoomendations to the state in terms of action that they ought to be 
taking. The areas that they looked at included the areas of curriculum, 
assessment of common curriculum testing; staff effectiveness including 
the areas of compensation, recruitment and staff development; technology 
and the effective use of time. 

The task forces worked for about four months and involved a 
considerable number of people in the state. They made recommendations to 
our state board and superintendent, with many of those recommendations 
being adopted. That moved us forward in terms of some action in the 
state. We view the actions as a continuation of something that was 
started back in the early 70* s when we established a set of standards for 
public schools that was orientated towards an outcome cased approach to 
education. Schools or districts were expected to establish goals for 
their programs. They would monitor students 1 progress towards those 
goals, and then look at their educational programs 1 effectiveness in 
meeting the goals and make modifications in programs based on the data 
related to those outcomes. They would also establish criteria for 
graduation in terms of student competencies. That program is still in 
effect but we have changed, perhaps, the role of the state. As Don Egye 
mentioned earlier, one of the things we looked at was what is the 
appropriate relationship between local districts and the state. What are 
the roles we need to play? One of the things we decided was that we need 
to have joint responsibility for the outcomes of education. The state 
haa responsibility for helping establish those outcomes. The first step 
was to move forward with a common set of goals in the curriculum area. 
That is consistent with the philosophy that we had before. It does, 
however, change the relationship between local districts and the state. 
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We have established a task rce of research and evaluation 
professionals in the state, to lock at where we are* We are in the 
situate ui that Washington was in. We have heen using these standards for 
awhile, but we don't have any data as to how effective we have been in 
the implementation of this kind of a program. We want to take a look 
Ottck at what ha* been happening and then we want to set some targets for 
1990 9 so we are trying to put those two things together in te.ms c* a 
research effort to help us look in both directions. 

As part of the Oregon Action Plan, we identified a series of 
/ariables that we wanted to establish as goals. The first was to improve 
st lent achievement across the state and this group has helped us 
identify some possible indicators. We are not finished with our task and 
some of these will probably be revised or added to. What we have, on 
student achievement is from our assessment results, but is fairly seager 
right now. We are trying to expand that in a way that doesn't cost 
money. If you have any ideas, I would be glad to talk to you later. 

The second consideration was improved community involvement and 
satisfaction with the schools, including the number of volunteer hours, 
citizen satisfaction and attitude, and employer satisfaction. We have 
some data on this, particularly or* employer satisfaction with students 
who have been involved in vocational education programs. 

The third area dealt with improved student success after leaving 
school, including high school attrition rate, job placement or continued 
study, and success in p. st- secondary education. 

The fourth was improv^J school climate which included student, 
teacher and administrator perception of the school climate, student 
attendance rate, teacher attendance rate, student behavior in terms of 
aspersion and expulsions. 

The last area is improved school productivity which we are finding 
difficult to measure. Ke have identified the cost per unit of credit as 
one possible indicator. 

In some of these areas we will gather data, while fo*. others the data 
is on hand. For sue of the indicators we will be satisfied with 
information from a sample of districts because these are statewide 
targets and we are looking at the impact over time of a statewide 
effort. From our perspective we need that information, but it is not 
necessarily information that is required from every school or from every 
student. So our approach will not necessarily be one of looking at it on 
a school by school basis. We are interested in statewide indicators. 

The second area of concern **; that dealing with the school profiles, 
and it has a different orientation or purpose. The task forces that we 
established looked at that issue. They said that it should address three 
areas: student performance, effective practices and programs and 
services. Then they said that it should minimally profile the following 
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areas: school and district philosophy; student performance data; use of 
effective practices; description of program services and personnel; and 
fiscal data. They also recommended that the profile should be uniform, 
statewide and updated yearly, and the information should be 
understandable to the public, should focus on school improvement and not 
school comparisons, and should be an efficient data collection system. 
That is a big task. 

We too?; that information and have been working on trying to establish 
our orientation to the idea of a school profile*. We have come up with 
the following definition and purpose. 

First of all, it should be a profile of summary information about a 
school. That includes data on program outcomes, processes and resources 
that provide the comprehensive description of its performance and other 
characteristics over time. 

Next, the purpose is to systematically monitor trends for school 
effectiveness, impact and productivity as the basis for informed policy 
management and operational decisions at all levels of the school system 
using multiple indicators of educational quality and equity. 

Last, the focus of the profile should be at the school level, 
requiring the committment of the school staff. The other levels of the 
school system should provide support. I want to spend n little time 
calking about this because I think it is an important part of our focus. 

The information that we have at a state level is very minimal. In 
fact, it is inadequate for the kinds of decisions that need to be made at 
a school level. We cannot gather all of the information that a local 
school would want in making decisions at a school level. We don't 
purport to have a profile that we would generate at the state level that 
would address the purposes we identified. So, it has to be generated at 
the local level. Effective change occurs when people who are going to 
make the change have a stake in the change. First of all, they have to 
feel that whatever it is that they are going to be dealing with is 
important, that there is a need in that area, and that they have an 
opportunity to influence the decisions that will be made. Then whatever 
change does occur, they will perceive that it is positive change, with 
data to back that up. We think those are important issues and we think 
those are things that need to be dealt with at a local level. 

Don Bgge talked about the Hoquiam Effect, I want to talk about the 
"so what" effect. Recently, we had a meeting where there happened to be 
a teacher (of all things) amongst all these administrators who got ur id 
said, "Well, what I'm going to talk about is something that I teach a_ 
my kids and that is to always ask the question 'so what'." We were 
talking about all these grandiose things and he was applying the "so what 
effect to them". If we are going to do this, the question is so wnat? 
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Why are we doing it? What impact is it gwing to have. If we ask 
ourselves that question and it drives us down to the realization that if 
we are going to make any change, the change has to occur at the school or 
classroom level, then everything we do ought to be geared to that level. 
It ought to be a system lat supports change at a school and classroom 
level. 

Xn developing our materials, wo made some assumptions. The first 
assumption was that we should have local indicators and in many cases we 
have only one indicator at the state level. Local districts are going to 
have to supplement this with othar indicators, and that is appropriate 
because they know which indicators they want to look at. In this state 
(I don't know how it is in other states) there has been, over a number of 
years, a lot of profiling activity or school improvement activity. We 
may not have called it that, but it was a systematic look at data on a 
program and looking at a number of different factors for a program and 
making decisions bases on these factors. I think we want to recognize 
that effort. This is not a new thing and we hope that our statewide 
effort will make it a more consistent effort throughout the state. In 
order to do that, you need to have multiple indicators and the profile is 
only one element in decision making. This was brought up earlier— that 
the information you have on paper may help influence the decision, but 
there are a lot of other factors involved. The profile will act as a 
catalyst for trouble-shooting and corrective action and Steve Slater will 
talk more about that later. My experience has been that if you have 
data, particularly if it is data that is important to people and you lay 
it out in front of them, they will deal with it. They will take action 
based on that data, but it has to be important data. It has to be their 
own data. The change over time is the best comparison, rather than a 
compari&in from school to school or caparison to a norm. 

In order to have a balanced set of data regarding a school 9 s program, 
will usually require the school to gather data on their own, Often this 
data may be less "objective" than data gathered from formal instruments. 
So, if all your achievement data related to, say, basic skills and that 
is only one area of your program, you may need to supplement it with 
whatever data you can get your hands on (perceptions of higher level 
thinking skills, or other areas in your curriculum) rather than to focus 
on only the one area. 

We set up a criteria for selecting indicators, including utility for 
decision making, sensitivity to change, commonality, validity and 
reliability and the feasibility of getting ahold of data understandable 
to lay audiences. 

We reviewed the information available in the computers and paper 
files of our State Department and tried to find what was there that we 
could put together for a profile. The logical way is to define what you 
want on a profile and then collect the data. But when you are starting, 
sometimes you have to resolve to use what you have and then try to put it 
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together in some kind of organized fashion. Here is what we came up 
with. First of all # how ©any students do we have at a particular 
school. In this state we have a lot of schools that have small numbers 
of students; we have sane schools that have less than ten students in the 
whole school and what does a school profile mean in that particular 
situation? There is a wide variability that can occur. So I think that, 
in this state, student enrollment is an important variable. We also have 
student attendance rate, and student mobility. For student achievement 
the only Measure we have right now on a statewide basis is eighth grade 
test results for the graduation rate and what we will give this first 
year is a state rate. We know what the state rate is, but we don*t know 
what each school rate is. We will provide a set of state data next to 
local data so they can compare where they are in relation to the state. 
We will include student ratio and ethnic composition. We will also look 
at the percent of students from families below th* poverty level (what we 
have there is an estimate from the principal) • Wo also have the students 
who speak languages other than English in the home. That is information 
we received in our eighth grade assessment which is based on a sample of 
the schools— we don f t have that for all the schools, ac well as for 
parents 1 education/occupation. Then we have the teacher experience and 
training, and teacher turnover. 

This is our beginning and we will be meeting with some people to talk 
about this entire process and looking at other data elements, but before 
we begin asking people for a lot of data — well, they have told us 
already, "Don't ask for a whole bunch of stuff and put us through alot of 
extra work if it is not going to be useful. ■ So the first thing we need 
to do is identify needs. If I ask for additional information from 
schools to use in a system like this, we want to measure it against this 
criteria, we want to be sure, in fact, that it will be useful information. 
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State Activities in Developing School Profiles, 
Data Bases and Indicators 

Stephen J • Slater 
Oregon Department of Education 



Today I would like to talk about a dilemma we have tried to resolve 
in putting together state-generated school profiles. In designing a 
school profile that the state could feasibly provide, we rather quickly 
came to the conclusion that it is hard to get good data on the factors 
that matter the mo*t to school effectiveness. It frustrated us that the 
state can only scratch the surface of tuat goes on in a school, A good 
example is the variable of academic learning time mentioned earlier this 
afternoon. The Department of Education can ask how much time is in the 
school day. We can even ask for information on how much time is 
allocated to a given instructional area. But, we can*t very feusibly ask 
teachers to. do the recordkeeping that gets r.S academic learning time. So 
we have a problem with the utility of state-generated profiles for the 
purpose of reflecting the variables in schooling that are most important, 
yet hardest to measure. How are we dealing with this problem? I would 
like to talk about two things. The first is Oregon's vision of school 
improvement. The second is the state's role in supporting school 
improvement. 

If there is one fundamental principle that has guided Oregon's 
approach to school improvement over the last decade it is this: 
meaningful and lasting efforts to improve school effectiveness originate 
fro© within the school system, as opposed to being externally directed or 
mandated. The state has recognized that commitment to continuing 
self-renewal and improvement is the hallmark of effective schools. 

The model of school improvement underlying Oregon's Standards for 
Public Schools and the Oregon Action Plan for Excellence calls for nine 
conditions to be met.* The model assumes that schools can and should be 
rational and accountable enterprises where: 

1. Instructional programs (such as mathmatics and science) are 
developed on a K-12 basis 

2. Goals and standards for student learning are agreed to and made 
explicit 

3. Classroom instruction and materials are appropriate for each 
learning goal and for each student 

4. Assessment of student learning is sensitive to (or aligned with) 
the goals of instructional programs 



*I am indebted to Ray Talbert, of the Oregon Department of Education, for 
his formulation of these conditions. 



5* Student achievement data and a broad range of related 

information are systematically gathered, reported, and analyzed 
to determine: the effectiveness of instructional programs, the 
extent to which the school is seen as a productive place to work 
and learn, and, the extent to which the outcomes of schooling 
expected by the community, district and school are being achieved 

6* Formal problem solving procedures are initiated whenever 
assessment results indicate that expectations for student 
learning are not being met 

7* The school staff has the ability to identify and implement 
changes that will result in improvement when improvement is 
needed 

8* There is a continuing effort to ensure the adequacy of the 
school's standards for student outcomes, and to keep the* 
school's curricula current 

9* The district is committed to providing the required resources 
and support, including a staff development program that focuses 
on the staff's ability to implement and operate this model of 
schooling* 

The essential concept embedded in Oregon's model schooling is best 
expressed by John Good lad. In the Foreword to The Structure of School 
Improvement (Joyce, et al*, 1983), he says that school improvement occurs 
when the process is established as part of the regular business of 
educational life* This principle is worlds apart from prevailing 
notions* It means, for example, that instead of seeking to install 
innovations, a school develops the capability to innovate* In effect, it 
becomes self-renewing* 

The major dilemma facing educators who work outside the classroom 
(e.g., at the district and state level) is how to create the conditions 
for effective teaching and learning without imposing excessive controls 
that stifle teachers' creativity or drain their energies in 
non- instructional tasks* Some believe that school reform can be 
accomplished by "tightening the screws" from the stata level—mandating a 
uniform set ct schooling practices assumed to increase st dent 
achievement* In Oregon, on the othor hand, we don't equate high 
standards with otandar ization of means and methods of schooling* We 
agree with Peters and Waterman, that effective organizations are 
characterized by the co-existence of firm central direction and maximum 
individual autonomy* Sxto organizations are unyielding on a few key 
values; in the case of schools the skills and knowledge children are 
expected to learn. But at the same time, effective organizations insist 
on entrepreneur ship and innovation at the front lines* Thus, the key 
question we have been wrestling with at Oregon Department of Education is 
how to mobilise delivery-level expertise and free up the creativity of 
teachers and building administrators? 
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arJZSl ii?, * ni 2* ?° ndition8 1 " s t ed wlier, a major ingredient in 
?nSr f f sch001 Movement is (1) the systematic monitoring of 

information on key performance variables, (2) analyzing such data to 
diagnose potential problems and (3) taking the indicated corrective 
action. Thesu three steps essentially constitute a troubleshooting 
o^;v?:^ Thfc , PUrpOSe , of troubl « s h°°ting is to diagnose the malfunction 
of an existing instructional program that is not working effectively. Or 
conversely, as the saying goes, "If it isn't broke, don't fix it." This 
approach stands in contrast to the attempt to import a new instructional 
program, complete with teacher-proof materials and hermetically sealed 
student activity kits. The latter approach may produce temporary 
learning gains, but ultimately, we feel, it de-professionalizes teaching. 

We believe the development of troubleshooting tecnriques in the 
schools holds promise for stimulating local innovation and problem 
solving. An appropriate state role, therefore, is to provide the tools, 
technical assistance, support, ar<d incentives for schools to effectively 
troubleshoot their instructional programs. The model we are pursuing is 
derived from the work of Ralph Vedros and Ray Poster at the Florida 
Deoartment of Education. They have developed a method of "situation 
analysis" which allows one to describe the circumstances operating in a 
school across four organizational levels. 

First, at the student learning level, situation analysis provides a 
description of student characteristics together with student performance 
patterns and learning expectations. 

Second, at the instruction level, factors are described, such as 
whether students are meeting entry requirements, are being sufficiently 
exposed to content (which gets at academic learning time), provided 
feedback and reinforcement, and motivated properly. A host of school 
climate indicators fall in this catagory as well. The instruction level 
also encompasses the description of teacher compentencies and working 
conditions in that school. 

Third, situation analysis at the management level includes job and 
authority specifications, resources, school organization, staff 
incentives, and provisions for staff development. 

Finally, the policy level involves an examination of the type and 
quality of guidance, supervision and regulation that take place at the 
district level. It includes an investigation of program coordination 
across school sites, allocation of resources, and program evaluation 
procedures. 

The process of situation analysis involves a series of probes, 
beginning with deficiencies in the student performance, and then moving 
upward to investigate the instruction level. As Vedros and Foster 
describe the process, first a set of hypotheses is formulated about the 
causes of low student performance due to the situation at the instruction 
level. These conjectures can be generated either from professional 
judgement or intensive analysis of student performance data. The 
hypotheses are ^objected to verification through a variety of formal ;,nd 
informal data gathering techniques (e.g., interviewing, observation, 



analysis of student records, use of surveys, and so on). Information is 
sought which either affirms or rejects the hypotheses, extends the 
hypotheses to include other causes, or links the cause with another at a 
higher organizational ievel. Basically, situation analysis is a kind of 
backward mapping process, beginning with student performance data, and 
then establishing the linkages between low student performance and 
factors that characterise the instruction level, management level, and 
policy or program direction level. This cluster of causes, together with 
the interrelationships among them, show the location of program 
malfunctions and provide insight into the redesign of program activities, 
procedures and policies. 

To sum up, in Oregon we are working on school profiling on two 
fronts. First, we intend to produce a relatively parsimonious set of 
indicators that would be common to all schools, providing information on 
trends in student performance, school characteristics and contextual 
variables. Wayne Meuburger, in the previous presentation, described our 
current thinking on this front. The generation of profiles that are 
common to all schools would in effect prime the pump, hopefully providing 
an incentive for local profiling. 

I want to emphasize that we don't intend or even anticipate that the 
state-generated school profiles would supply all the information 
necessary for good program improvement decisions. Thus, our second focus 
is to support the development of troubleshooting techniques at the school 
and district level, primarily through assistance in local profile 
development and the analysis of data from multiple indicators that are 
selected locally. The Department of Education can provide a variety of 
instruments and models to assist in the situation analysis process 
described earlier. We feel that to do a good job of problem solving 
schools need a broader range of information than the state can provide. 
This information must be sensitive to local needs. A by-product, by 
virtue of the fact that a school selects its own indicators, is that the 
threat potential of profiling is dramatically reduced. There would be no 
schools with the same data sets, reducing the threat of interschool 
comparisons. 

Our ultimate goal is to empower the school, through the careful and 
deliberate analysis of outcooe process and contextual data. As Goodlad 
stated, our intent is to make school improvement a part of the regular 
business educational life. 
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The High School and Beyond Database 
Local, State, and National Perspectives 

Dennis Deck 
Northwest Regional Educational Laboratory 



I an going to briefly describe a da abase that has applications at the 
local, state and national level. The High School and Beyond Study provides 
an interesting look at the educational and occupational plans and activities 
of high school students. The Department of Education, Congress and many 
researchers have made use of the extensive database generated by this study. 
The State of Washington has piggybacked on the national study to create its 
own database and made a number of decisions based on those results. The 
Northwest Regional Educational Laboratory has been working with districts in 
Washington and Oregon to make local use of it. 

The High School and Beyond study was initiated by the National Cent 2r for 
Educational Statistics (NCES) in 1980. Based on the National Longitudinal 
Study that was conducted from 1972 through 1978, the study looks at the plans 
and activities of high school sophomores and seniors with a follow-up of 
second year graduates. The study looks at both the education students are 
getting and their educational plans, their occupational plans and how 
students are preparing for their occupation. 

The comprehensive database generated by the study includes ability test 
and survey data from about 58,000 sophomores and seniors in the first year. 
There were also school, teacher and parent surveys linked to the student 
responses. 

The design was longitudinal, designed to follow-up the same students at 
two year intervals. Sophomores in 1980 were included as seniors in 1982 and 
graduates in 1984. Actually, little has been published about the 
longitudinal results of the study. Most studies and NCES reports have 
focused on the difference between the 1980 results with the 1972 National 
Longitudinal Study results. 

I want to focus on tne senior survey itself because I think that it is 
the core of the study. From the state and local perspectives it seems the 
most useful. The survey asks seniors what they are going to do when they 
leave school, what job they think they will have, and what work experience 
they have currently. It asks whether the student is academically bound or is 
in a vocational type program and what courses they plan to cake. It asks for 
detailed information about the school program, about support from the home, 
about self or school attitudes, about further educational plans 'either 
college or national), and abou the student's anticipated vocation. 

Now I will touch on some of the national findings that show typical uses 
people have made of the data. Between '72 and '8u, studies have revealed an 
inflation in grades. These studies also noted a decline in the homework 
students were doing. More recently published data shows that there is a 
discrepancy between opportunities in the job market and what students were 
actually planning to do* Let me show you part of that. As you can see on 
the transparency, the dark bars represent opportunities in the labor force 
and the shaded bars are what seniors want to do. You can see that the 



professional, technical skills are the glamorous jobs, the jobs kids say they 
want, but clerical fields, the crafts or trades, and the services are were 
the jobs aie. So, there are some real big discrepancies here. I just wanted 
to show this as an example of one national finding that has come out of the 
National study. 

The State of Washington has made much use of the High School and Beyond 
database for two reasons* First, they wanted to have reliable data for state 
level decision making. Second, they wanted to have a database for local 
comparisons. They wanted local districts to have representative state data 
for self-comparison. During the national study in 1980, they had National 
Opinion Research (NORC) , the High School and Beyond contractor, over sample 
in the state of Washington so that there would be enough students tested to 
have more reliable data. In 1982 they did their own follow-up because they 
found that NORC 1 8 services were getting expensive. 

The state data, like the national data, in available for independant 
research projects by district*, universities, or other agencies. You can 
contact the state of Washington if you are interested. 

I think there have been some interesting uses of the data in Washington. 
One is that the state board has increased course requirements to promote 
better planning by students. Tue state found that even the better students 
wtje not laying out appropriate plans given the college or occupational plans 
they reported. For example, one-third of the students who said they wanted 
to go into engineering in college were not planning to take calculus. The 
High School and Beyond data also affected the states decision to add eighth 
grade statewide testing. More recently the state has added High School and 
Beyond items to the 8th and 10th grade statewide assessments. Thus, now 
there is a student attitude supplement to the statewide achievement test. 
Participating districts will have both local and statewide data and district 
data on selected High School and Beyond survey items. 

My involvement with High School and Beyond has been primarily at the 
local level. We have put together a scoring service so that districts can 
administer the High School and Beyond survey and we at the Northwest Regional 
Educational Laboratory will analyze those results and provide summary 
reports. The district administers the survey, which it printed as a reusable 
booklet. Students take about 30 minutes to respond to the 130 questions on 
the rurvey. The district sends the answer sheets to us, we score the answer 
sheets and provide reports. The reports show the responses to each item and 
a series of cross tabulations compare two variables. For instance, some 
districts were interested in sex differences, so key items were 
crosstabulated by sex. Other districts requested crosstabulations on student 
performance variables like grades and homework. We also send an 
interpretation guide which includes 9 for each item, the state and national 
breakdown of student responses. So here we have comparative data from both 
Washington and the nation. 

Many of the districts that are using this service now are involved in the 
Onward to Excellence (OSE) program. As part of the OTE school improvement 
process, schools annually develop profiles to help target weak areas needing 
imporvement* They are using selected items ft am the survey to do this. As 
Oregon districts get further along in trying to come up with school profiles 
O directed by state requirements; they will find the database useful as 
ERJC Jk A.||jw ^itjtei^ evaluate specific programs like one 



district here in Oregon that wanted to take a look at its counseling 
program. They selected certain items that reflected the kind of help 
counselors were giving students and student's reports about how helpful the 
counselors were. Other districts using our scoring service simply want to 
take the pulse of the district, to see how well they are doing. 

A couple districts, Seattle Public Schools, for instance, have also used 
the freshman/sophomore surveys heavily. This survey is more useful in 
counseling individuals or small groups early in their schooling, rather than 
program evaluation. 

Let me show you the school profiles from two schools involved in the 
Onward to Excellence process. In the first transparency, the question is: 
Should there be more emphasis on basic academic subjects? The percent of 
students responding positively to the question in this school was quite 
different fro© the results for both Washington and the nation, but the 
percentage did decrease across years. This finding was used in goal setting 
for the program improvement project and the results suggest that there was 
some impact of the program efforts. 

The other school was interested in "what factors interferred with your 
education.* As the tr sparency shows, they found poor study habits were 
something that the stu< its were co erned about. 

One thing I really remark about the data is how close many of these 
schools are to the national results. I expected much more variation across 
schools and less stability in the data across years, that would make the data 
more difficult to interpret. That is not what I have been finding, where 
there are differences, there is usually a good reason for those differences. 

Let me summarize the benefits and problem* with using the High School and 
Beyond survey to create a database. On the positive side, the High School 
and Beyond data supplies a wealth of information about student plans and 
activities that is a supplement to achievement data. Educators have trouble 
finding good measure or indicators at the secondary level beyond test scores 
and dropout rates. 

The survey provides comparative data, national, state or local results 
from previous years, with which to put the student responses in context. You 
must have some basis for interpreting the numbers, they are not very 
informative otherwise. The lack of comparative data will be a problem for 
Oregon districts as they begin to build and try to interpret school 
profiles. Without data from other districts or data from several years, it 
is difficult to interpret most indicators, to know what is a high or low 
value. This is true in any discipline, not just education. 

On the negative side, these are self-report data. NCES has supported 
national studies that show that the student responses to these items are 
suprisingly valid. However, there are limits to what you can show with 
student self-report data. I think that we want to include additional 
measures for a more complete set of indicators in our profiles. Also, keep 
in mind that the comparative data being used now is based on the 1980 
original survey and so it is getting a little out of date. 
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The information in this presentation is the result of discussions of a 
working group of representatives of majo^ information system centers in 
the Northwest Regional Educational Laboratory (NWREL) region. The group 
was convened to consider the feasibility of regional databases. The 
dictionary definition of feasibility indicates a practical interpretation. 
That is, a feasible activity is capable of being accomplished, is 
practicable, suitable, or logical. The discussion focussed on those 
issues of practicality, after the group first dealt with characteristics 
of databases and reached consensus on a definition. 



A database is useful when it answers the questions people ask. It is 
certainly possible to identify information that is regional in nature 
(that is, it will transcend s+ate boundaries), and it could be collected 
and organized. However, if it is to answer questions , we need to answer 
other questions such as who will ask for information, why will they ask, 
and what data needs to be present. 

Who will ask? The following list shows some of the client groups in the 
Northwest states who possible have common interests in certain kinds of 
data. 



Introduction 



What is a Database? 



A database is a useful collection of data which: 



• has a purpose; 

• is organized; 



• is valid and reliable; 

• is accessible 



Chief State School Officers 
SEA staff members 



Higher Education Systems 

Legislators 

Governors and staffs 

Lobbying Groups 

Private sector — publishers, 



Regional Centers or Co-ops 
School district staff 



Boards of Education 
Professional Organizations 



photographers , etc. 
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Why will they ask? The major reasons identified by conferees for queries 
ejiswerable by regional information can be grouped in three categories: 
comparability, planning, and justification, 

a. Comparability-school districts, state agencies and other 
organizations frequently wish to compare themselves with 
other agencies of the same type and size, particularly in 
the areas of budget and policy, some maintain a phone list 
of comparable agencies, polling them as needed. 

b. Planning — many agencies need to look at their cwn planning 
in terms of regional trends. 

c. Justification — most education agencies need data to "build a 
case" for proposals to boards, legislative commitees, and 
the public. 

What questions will be asked? A number of different categories of 
information were identified as the subject of inquiries to data centers 
in the region. A sample of these is below: 



The Common Core Data (CCD) is a database provided by states to the 
National Center for Educational Statistics (NCES) each year. Some of 
the above data is included, and can provide a good <*tart at a regional 
database. Subsets of the national collection can be obtained by state 
and region. 



Attaining these characteristics in a regional database could be difficult. 
The larger the area covered and the larger the scope of data, the more 
difficult is the task. However, some steps to achieve higher degrees of 
validity and reliability can be identified. 

a. Achieve common definitions of terms — many terms in the 
education field acquire a variety of meanings. The 
terms Basic Education, Dropout and Middle School are 
examples. Sometimes meanings are arbitrary numbers or 
levels set by boards or are defined in laws or policies. 



Student retention 
Teacher supply and demand 
Certification 
Teacher assignment 
Student assessment 
Teacher /Student ratio 
Information sources 
Studies and reports 



Policies 
Budgets 
Curriculum 
Goals, Objectives 
Nonschool data 



Tax 

Income 

Population 



Validity and Reliability 



ERIC 
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b. Insure complete participation — a database drawn from many 
agencies is as complete as the degree of participation of 
the agencies. The regularity of both contribution to and use 
of data affects validity and reliability. The dedication to 
detail of the contributor also is important. 

c. Insure currency — frequent and continuous update, on a regular 
and prompt basis , is important. In a comprehensive database, 
there will be great variety in the periodicity of reporting 
data items , ranging from monthly to yearly. 

d. Establish good management and liaison— continuous conx ict with 
the contributing agencies is required to carry out the previous 
steps and maintain the efforts. 



The characteristics of good accessibility were identified as be), 
direct , achieving timely response, ease of requesting information and 
ease of update. A major characteristic is the ability of a database 
system to respond to a reasonable ad hoc request. Many computer-based 
systems have standard reports but do not support easy retrieval of 
information in any other fashion. 

Implementation—databases are typically collected and stored either on 
paper or in computers. Sometimes paper systems are more accessible 
than computers. However, the larger and more complex the database, the 
more difficult is the paper method. 

Location — a database does not necessarily need to be in one file or one 
computer. A regional database might be conceived of as a collection of 
databases in each state, with access enhanced by telecommunications 
systems. In the NWREL region some major databases exist in computerized 
form. The following organizations are examples: 



Cost—cost factors such as af fordability, cost worthiness and cost 
effectiveness affect access. If agencies can't afford it, they won't 
access it. 



Most conferences agreed that a regional database is probably feasible. 
That is, it is possible to obtain data, organize it, provide access, and 
maintain reasonable validity and reliability. The question of cost, 
particularly affordability, needs further research. While a center like 
OTIS currently serves its clients with similar services in an affordable 
fashion, it is not clear how much a region-wide service would cost and 
whether education agencies would pay the cost. 



Accessibility 



Oregon Total Information System (OTIS) 

Washington ESD Consortium 

Idaho Department of Education (S SPARS) 



Conclusions 




Using Data Systems to Support a 
District's Evaluaticn and Improvement Efforts 

William Cooley 
University of Pittsburgh 



I work at a research and development crater at a university. Maybe I had 
b#tier say that I am paid by an R & D center at a university. I work for the 
Pittsburgh Public Schools. NIE, in its wisdom, gave us funds to support a lot 
of different kinds of learning research. What I chose tc do with my share of 
that is to use the Pittsburgh Public Schools as the laboratory in which I 
explored what district-based R&D efforts—at least research/testing/ 
evaluation enterprises — might uo to be more responsive to the needs for 
information on the part of school *x>ard superintendents, managers of schools, 
teachers, etc. 

Bill Bickel and I started this work in 1978, and we have been at it ever 
since. The big change came in 1980. We got a new superintendent of schools, 
Richard Wallace, and he is a very data-oriented superintendent. He looked 
around. <Ia noticed he didn't have a research office. He found us across the 
street, and we started working very intensively with him. Now we have a book 
coming out next month, entitled Decision-Oriented Education Researc h, to be 
published by Kluwer-Niejhoff The benk is about how to do district res arch, 
evaluation and testing. Now, there ars other people who are doing it as well 
as we were, but they don't have the advantage of extra resources to reflect on 
it, to write about it, to tell the others what they are doing. We have had 
the luxury of «*eing able to reflect on that, and also to document our 
reflections. We have been able to actually study the use that the district 
made of whatever information we produced. 

We have 11 case histories: what did the district ask us to do? What did 
we do? What happened? How did we go about doing it? Wha* did we learn? 
What did the district do with what we learned? Then we went acros these case 
histories and tried to generalize. The generalizations included * ;gs like 
what client orientation is all about, why it is critical, and lo* to keep from 
"being used" in your anxiousness to be useful, which is an important 
distinction. So what we did was develop notions about "client orientation", 
notions about the importance of being methcdlogically eclectic, notions about 
formative— that is, improvement-oriented— ctudies rather than summative ones. 
I get very uneasy, as I did y^fc^day, when aix of the emphasis is on 
summative evaluation. Laying on yardsticks to pass judgement over what must 
be going on—that tends not to be too useful as people try to figure out how 
to make things better. So our emphasis is very much formative in nature. 
Larry Barber apparently has ; <st come out with another one of his rash 
statements in which he has "proven" that formative evaluators tend to be 
optimists and sv^&ative evaluators tend to be pessimists. I like it. It does 
bring out the optimist in me, as we seek to figure out how to make a messy 
situation more workable. So, tba^s what Decision-Oriented Educational 
Research is all about. 
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One point we make—and this won't be new to you folk*-- is that most of the 
effort people make to generate information which others in the district will 
find useful is in putting together the data that are likely to ^hed light on 
some issue* And since these issues have a way of changing very quickly, you 
have to be timely. If you set out to do a two-year study every time the board 
lias a need to understand something, you're not going to make it in this 
business. Now you all know that, but what we did was to try to figure out 
what to do about it. Clearly, one aspect is to put together an information 
system that allows you to have the kinds of data that are needed for most 
questions that come up, but have to be ma saged in different ways to guide the 
dialogue as a board deliberates, or guide the manager who is trying to set 
priorities. So we have been figuring out what such information systems might 
look like. That's my current distraction. 

The distric as a central computer— the main frame that sits in the 
central office. he district has grand ambitions in terms of using the 
computer to serve people in the schools. It's had this computer for five 
years, and it never gets to the part about serving people in the schools, 
because it's distracted with central office concerns. The guy down the hall 
has to get out the payroll, or the bus schedule has to get worked out. 
Noticing that, we began to think about what might be an alternative, and we 
began to get interested in this thing called distributed processing. But 
another event in the district also propelled me into looking at local school 
information systems based on a microcomputer. That phenomenon is the 
principal's function as an instructional leader — the pressures on principals 
to be well-informed about instruction in the schools, to produce high scores 
on achievement tests, etc. With these pressures, we began to notice some very 
weird things happening with the indicator called the California Achievement 
Test. That's toother interesting thing about indicators: they are easily 
corrupted^ especially when used for summative evaluation. So I was asked to 
talk about why I didn't think achievement tests were a good idea for 
evaluating principals, and I laid out some things. What we had to do was to 
help the board see why they shouldn't fire a principal if the CAT scores were 
low, why they might start to get strange looking miracles in certain schools, 
and what it means to administer standardized tests in nonstandard ways. So we 
asked some questions, particularly the second one, Do principals have adequate 
resources to monitor and correct ir,**fe<jtive teaching? (See page 3 of the 
attachment to this presentation.) The answer was a clear no. They had 
mounted a number of important innovations in the district. There is in fact 
in this current issue of Educational Leadership a nice article by 
Paul LeMachieu on Monitoring Achievement in Pittsburgh (MAP) , which is our 
criterion referenced testing system. 

As I began to think about the principals' needs for information, I began 
asking them what they were getting out of the central conputer. And they said 
well, about the only thing we get now are memos from central office at the 
speed of light. And so, electronic mail was the one thing that they 
noticed, but they were not getting any help in putting all this stufi 
together. So we oegan to think about how principals might inplement the 
instructional leadership function. What can a principal manage? Well, the 
principal manages the schedule, grouping, staff development, staff deployment, 
learning resources (page 1 of the attachment). Those are some of the 
manipulable things. We also looked at th<* constraints— and there are a lot of 
them— all the way frcm school board politics, »te ?ode, local conventions, 
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AFT, the union regs, etc. Then we asked, what are the options, what are the 
restraints, what kinds of things might they monitor? And we came up with this 
familiar list (page 2), And we began to notice that, without too much 
trouble, we could get a lot of this in one place, making it possible for the 
principal to look across students and across classrooms anrt monitor indicators 
in a way that allows for corrective action. 

That's my enthusiasm for indicators. I think the most important function 
of indicators is to look at indications of outliers or trends or whatever, but 
have the next step be conceivable, be possible, be implementable— we call them 
■action systems, • It's the old cybernectic model. You notice that something 
isn't the way you would like it to be, and you have procedures that can be 
used to follow up. Those procedures are necessarily diagnostic, because it's 
never clear from the indicator just what the problem is. All it tells you is 
that maybe there is something you might want to look into. So, we had this 
model of indicators as guiding corrective feedback—a formative orientation. 
He had the notion that principals needed more help. And we began to design a 
system a little over a year ago. He called it Computer-Assisted Principal-- 
CAP. He put a lot of effort into designing a nice, pretty logo to look at on 
the monitor, but we began to notice that a lot of other people besides the 
principal were using it. So in order to save the logo, we call it Computer- 
Assisted Professional, (See page 4 of the attachment to this presentation.) 
It is designed to be used in the school, by the people in tho school, for the 
people in the school—their very own data base management system. 

Now you may ask, what does this have to do with the notion you started out 
with—that if you are going to get good data centrally, something has to be 
different. Hell, the only way I know of having good clean data— data that are 
reliable, valid and so on— is to have people looking at those data every day, 
using them, and caring for them. The way to get that behavior is to build 
incentives for them to use the data, to get data in there, and to get 
information out. If people are using the data every day in the trenches, then 
summaries of those data can be passed up to the district and used for 
planning, etc. I am a very much of a bottom-upper, as you may tell from the 
way i am talking. I think it's obvious that certain functions have to be done 
at the state, but I get very upset when the state gets involved in monitoring 
student level achievement and is not noticing that 10 percent of the 500 
school districts in Pennsylvania are going bankrupt. There are certain things 
that they might monitor— things that they can do something about— and that is 
the point. You monitor the things you can do something about. If yon monitor 
those things that are your responsibility, you don't monitor the ones that are 
somebody else's responsibility. That's what we are trying to figure 
out— whatis the principal's responsibility? Hhat is the social worker's 
responsibility? Hhat is the teacher's responsibility? Then, we try to make 
displays of d*ta Miat are useful to them as they think about where they are 
and where uiey *re going. 

He designed this thing with some goals in mind, starting with very general 
ones. He deliberately set out to work on these three goals (page 5 of the 
attachment) • 




You all know about the first one* One of the things that makes elementary 
school life so difficult is the restriction of the schedule— the inflexibility 
of the thing, so, one of the ways in which a microcomputer can enhance the 
quality of life is through making life a little more flexible. In terms of 
equity issues, one of the very important functions that this system is 
performing is in improving communications between home and school. That is 
very much related to the issues of truancy, and we have become aware of a lot 
of reasons for kids not being in school that have nothing fco do with their 
attitude about school. He thought about goals for the system, and we thought 
about functions* One of the things we looked at was the district's Monitoring 
Achievement in Pittsburgh (HAP), their criterion referenced testing system, 
which is a period testing system. The district issues tests four to six times 
a year in each of four or five subject areas. Those tests are administered, 
collected centrally, scored centrally, and, if everything goes well, in two 
weeks they are back in the classroom with a report. What we noticed was that 
if we put a desk top computer onto an XT, w? could have the scores that same 
day. So, one thing that we built into this thing from the beginning is the 
ability to score tests. If you think about these microsystems simply as 
scoring devices, which they can very easily be now with the fairly 
economicaldesk top scanner, that opens up a whole lot of possibilities for 
entering data. A big problem in these microsystems is figuring out how to 
enter the data. One way is to score tests and to do attendance accounting. 
We do both of those, we also designed our own attendance system and Mve it 
operating on a daily basis with daily reports. Then there are the obvious 
functions of record keeping, record retrieval, report generating, and so 
forth. I could tell you stories about how each of these works, but bottom 
line for me is getting corrective feedback— organizing these data in ways that 
suggest next step?, or at least the possibilities fcr next steps. That's 
where we are trying to go with this sy*~.*tju 

Now, given the goals and given the kinds of functions we want to perform, 
what would it require? (See page * of the attachment.) We thought about this 
and made some decisions. Ke wanteu to build on the fact that, today, you can 
buy very exciting pack^s for managing data bases. A year ago Septemirsr, we 
looked at the 64 data base packages available for DOS . The one we picked was 
particularly exciting, because it has both the procedural language—that is, 
you can easily write programs that generate menu» and make a ve*y 
user-friendly system—and is as open ended, which means that any inquirer can 
make any ad hoc inquiry— a capability we have found to be extremely 
important. We went with a relational data base management software called 
•Knowledge Manager 1 — called it , K-Man l for short— and it is an excellent piece 
of software. We also knew that we wanted to get data into and out of the 
system very easily. So we went with the Hayes telecommunication software and 
internal modem, and that has proven to be extremely valuable, when the 
standardised test results come from the California Achievement Tests to the 
district's central computer, the file3 are downloaded for tha\. school at J they 
are off and running with the test data— not the way they were organized when 
the kids took the tests last spring, but according to the way kids are 
currently organised in the school. We wanted to be able to impoit data from 
the scanner, so that defined certain conditions. :;e wanted it to be menu 
driven, and that helped us define what data base system we wanted. We wanted 
graphics displays. I am a great believer in the notion that f one picture is 
worth 10,000 numbers 1 and I think that that was a good choice. You spend a 
little bit more for graphics, but it is well worth it. You also buy color 



which is really kind of fun. We also noticed that the district wanted to send 
letters home, pulling data out of the computer, and including them in 
individually tailored letters for each prrent. The first time we did it, it 
was a 12-hour run, and we quickly upped the priority for fast printers. And 
though we didn't start out to build a system, what was available just didn't 
make any aenoe, so we moved quickly to the hard disk and we think that that 
was an extremely important decision. The success problem is interesting: at 
first; we were wondering whether anyone was using it, and now we are wondering 
how we are going to handle the queue, and so we are starting to look at ways 
of solving that. 

Essentially, we are talking about building a school-based information 
system on an XT-like device at a school, for a school, with the people in the 
school guiding its development. That latter is an extremely important point. 
We began with the largest elementary school in Pittsburgh last fall; it has 
over 800 students. It was a mess in terms of information. Nobody knew where 
anyone was. The first time we did on alphabetical listing of all the students 
in the school, the principal cried. It was just a whole new life, and it's so 
such fun to see them moving into the 20th century. We worked with them 
veryclosely. Last year in our development school, we worked primarily % : ith 
the principal, the social worker, the front office, the vice principal—and 
the system took shape based upon that. Now the reason we worked with this 
group was that we knew we had to get their attention; we had to have them see 
that they could get more than they thought possible once they took the trouble 
to keep this thing current. So we started with central functions— the central 
office functions for that school. That proved to be a good idea; but we soon 
began to notice that we were making the same kind of errors that the folks at 
the central computer had been making — we found that focusing on management 
functions is not necessarily directly relevant to instruction. 

This year we are in another school which Pittsburgh has just launched, 
called Brook line Teacher Center. Some of you may have heard of the Schenley 
Teacher Center; it has been written up a lot. If you haven't heard of it yet, 
you will. I think it is a remarkable staff development effort. We took one 
of our high schools, closed it down, and reopened it with the best secondary 
teachers in the district. At any given moment, tive percent of the teachers 
are in the Schenley Teacher Center for intensive two-month inservice. The 
Schenley Teacher Center is a remarkable effort in a district determined to 
mount an intensive effort at staff development. They are now setting up a 
comparable center for the elementary schools, called Brook line. We are there 
in an environment with three times the teachers normally assigned to an 
elementary school. That is exciting, because you can finally talk to teachers 
during school— and that is unheard of. We have worked with them since 
September, and we are getting ideas from them about what this system might 
do. One of the things I am noticing is that the special teachers — such as the 
speech teacher, who deals with 50 of the 500 students in the school— want to 
build their own little data bases. That is, we are noticing what they want, 
trying to deliver it, and then observing what they do with it. 

What I think I will do is c throuyh a couple of the menus just to give 
you the flavor of it. The main menu (page 8 of the attachment) indicates that 
one of the things you can do in this system is monitor achievement at the 
student level. You can also monitor achievement at the classroom level. 
Classroom management data are very useful when the third grade teachers want 
to sit down and talk about what is happening in third grade reading. And we 



have data on student hone room and home data-* that file was established 
immediately by down- loading everything the central computer had on kids' 
backgrounds. The data in central computers have tended to focus on 
demographics, because central office staff have been concerned with such 
things as student socio-economic status. That kind of thing was down- loaded 
quite easily from the central computer , and it provided that data base. There 
is information on staff schedules, which in this particular setting of the 
Brookline Teacher Center, is important because of their need to schedule 
teachers together for inner vice. 

Then there is my favorite, which is ad hoc inquiry. That is always a menu 
option, and it is part of the reason we bought into Knowledge Manager, "K-Man" 
for short, with K-Man, I can teach a teacher in ten minutes how to do an 
ad hoc inquiry. They need to know about five commands, and that's it. They 
can then do their own inquiries, generate their own reports with the data that 
they want to see together, and it is very easxly learned. He use telecommuni- 
cations when we are passing data from micro to aicro, or from micro to 
central, or central to micro. Scoring tests involves not only scoring the 
math test, but also scoring teacher-made tests, when we score the math test, 
we capture the results, and it is immediately part of the student's record. 
Tteir aggregate is immediately part of the classroom record, and also 
produce at that time the reports that the teachers and principals want. In 
addition to having it become part of the permanent record in the file, they 
produce immediate reports. Now, we score but we don't capture teacher-made 
tests because they are not very systematic. But we are trying to figure out 
what things might look like if they did want to capture and build up their 
own file. 

Attendance accounting is a bread-and-butter kind of function. This is 
er^remely important, because money and other things are at stake. We are 
doing that now, and this is the first time the district has had computer 
support for attendance accounting. It just hadn't been done. It was never 
possible to learn what was happening at the student level in attendance, and 
we now get a daily attendance report, a 20-day summary, and data on the worst 
cases— a list that goes automatically to the social worker so that priorities 
can be influenced by the worst and the toughest problems. 

Let's say you picked student achievement monitoring after the main menu. 
It then reminds you that the files are structured by grade level, and you then 
indicate which $rade level you want to look at (page 9). Let's say you pick 
fourth grade, then it tells you what is available for fourth grade (page 10) . 
Some of these are mysterious at first. For example, predefined reports, 
doesn't tell you much, except that you might guess that they are reports about 
student achievement monitoring that people want frequently. Then a menu 
follows it. It is a very easily learned tree structure. You have all seen 
them, so I yon't go into it. 

If you go to predefined reports or you're interested in noticing 
relationships between standardized tests and criterion referenced test results 
in *ays that this last fall you just reassigned kids to home room based on 
last spring's CAT results. You know there is something funny as a result of 
the first math test in MAP. Por the first criterion referenced test, you 
notice some discrepancies. You want to look at it more systematically and up 
on the screun comes h scatterplot for fourth grade CAT and MAP— this in terms 



of standardized percentiles, this in terms of percent of items correct. (See 

page 11.) These are for the 70 sane students who are in level four on the MAP 

test. What we are trying to do is finJ out if people in the schools really 

understand scatterplots? How do they feel about graphs? what kind of graphs 

■ake sense? How do you notice outliers, which is one of the things we are 
trying to get them to do. And outliers like this— we find that the people can 
pick that up very quickly. ^ * 

Say you go to classroom achievement monitoring (page 12). As a part of 
fostering professionalism among teachers, one of the things the AFT is 
encouraging and we are encouraging is that teachers sit down and look at data 
together. It is amazing what occurs when a principal, teachers, the 
instructional leader, and maybe the math supervisor sit down and look at data 
together, try to figure out what the instructional implications are. The data 
can then inform the dialogue, which is all data really do well. They are 
helpful for getting some discussion going and for helping to clarify 
discussion. They don't make decisions. Thus, if the user picked option 4, 
under Classroom Achievement Monitoring, they would get a series of graphic 
displays that look like this. Now (see page 13), in this particular school 
there are chree homerooms per grade level, and rhat we are showing for the 
first ten math objectives is the percent of the students in that classroom 
who have mastery on each particular objective. 1 is is at the beginning of 
school. We are also capturing whether the objecti ea have been taught or not, 
because the pacing could be quite different from h*.neroom to homeroom. What 
we are getting the teachers to notice is whether the results say something 
about those particular kids, or about the teacher him/herself, or whether the 
results are common to all the homerooms, or if they point to a district-wide 
phenomenon that this particular objective is unusually difficult? Does it say 
something about the curriculum? you are trying to bring a lot of different 
data together to sort them out. The trick is to have a discussion with the 
teacher in question without getting that teacher all defensive. We are trying 
to get them to look at data that aren't summative, aren't being used to pass 
judgement. Instead, the group members are functioning like doctors who are 
looking at the X-ray and trying to figure out what to do. That is quite a 
different way of thinking about data than the approach people often have. 

I am in a little hot water in the district for creating a demand for 
something on the part of 90 other schools. That is encouraging for us, but 
disconcerting for my collegues in the district who are stuck with the 
implementation of this. But they are now planning distributed processing in a 
very serious way, whereas they weren't before. And principals are corncerned 
because there is a great deal of pressure on them to do a lot of things. The 
superintendent is very determined to get principals aware of and involved in 
instructional planning, and they need data for it. They need to be informed. 
And they see this as a way of helping. So we will probably be working out a 
schedule of implementing this during the next year. 

The King School, where we started last year, is a special case this year. 
We abandoned it. And it was deliberate. We developed the system there last 
year, got them well trained, and then said we were going to move on to 
Brookline where we have a chance to talk with the teachers and figure out how 
we might make a system useful for teachers. So we are now watching what is 
happening in the King School. We now notice that there are some problems, So 
far the main problems have been fights between the central guy, who said to 
the clerk, "You're not keeping my central computer current, you're spending 



all your tine on Cooley*s computer, and you work for me." So we are trying 
figure out how to resolve some of those little problems. There is some 
anxiety on the part of the Central Office people as they begin to Bee that 
distributing information is also distributing other things— things such as 
power. We have got some very interesting problems to work out there. Also 
when you distribute information, distributed processing is also distributed 
technical headaches. We are keeping track of those things—the number of 
times it goes down because the paper jammed or because the power surge didn't 
work. We have had a couple failures. The questions are how often does it 
happen? What do you do about it? How do you handle backups? Things like 
that. It keeps us off the streets! 
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What can a principal manage? 



Schedule 
Grouping 

Staff Development 
Staff Deployment 
Learning Resources 



ERIC 
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What can a principal monitor? 



Standardized Achievement Test Results 

Curriculum Placement 

Pacing 

District Test Results 
Curriculum Test Results 
Att&iuSsxtCG 
Discipline Actions 
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Can a Principal be Accountable 
for Student Achievement? 



•II 



- Can Initial student differences be taken 
Into account? 

- Do principals nave adequate resources to 
monitor and correct Ineffective teaching? 

* 

- Can principals keep ineffective teachers 
out of their building? 

- principals have adequate options for 
dealing with disruptive students? 
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Computer Aarirtad Pirofeariona! 



A school-based management information system 
developed fcy the Eyafaatioo Unit at the 
Learning Research and Development Center 
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What are the Goals? 



Improve student achievement 



Enhance quality of school life 



Provide equal opportunity to learn 
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Current Functions 



Test Scoring 
Attendance Accounting 
Record Keeping 
Record Retrieval 
Report Generating 
Data Analysis 
Corrective Feedback 
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System Requirements 

Relational Data Base Management Software 
Telecommunications Software 
Import Data from Scanner 
Menu Driven 

» 

Graphics Mc ltor 
Fast Printer 
Hard Disk 

Allow for Multiple Users 
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G&P MAIN MEFW 



1 . Students Me Mmm aat Monitoring 

2. Cl'^ssrocsB AcStit&soBUBte, Monitoring 

3. Student BtaDMSEQconi anxfi Bone Pate 
Staff and Schedule Information 

5. Ad Hoc Inquiries 

6. Telecommunications 

7. Scoring Tests 

8. Attendance Accounting 
9- Ea&dfcap CAP Data Files 

OL End GAP Session 

Enter tte Bfttober of Your Choice: 
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Which Grade Level Do You Want? 

0. Kindergarten 

1. First Grade 

2. Second Grade 

3. Third Grade 

4. Fourth Grade 

5. Fifth Grade 

6. Special Education 

7. Early Learning Skills 

8. Demonstration Data 

Enter Number of Desired Grade Level 
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Student Achievement Monitoring foi Fourth Grade 

1. Predefined Reports 

2. GrapMcs Displays Created from MAP Data 

3. Look at IndlTldual indent Achievement Records 

4. Look at individual Stadent Reading Progress 

5. Ad Hoc Inquiries 

6. Edit Student Achievement Data 

7. Choose Another Grade Level 
0. Exit Back To Main Menu 

Enter Number of Your Choice: 
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Scatterplot foa* Gr*a*le 4 CAT and MAP 
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Classroom Achievement Monitoring 

1. MAP Objectives Not Mastered 

2. Graphics Displays Created frak MAP Data 

3. Look at Individual Classroom MAP Records 

4. Percent of Students Passing Each MAP Objective 
6. Ad Hoc Inquiries (KMAN Prompt) 

0. Exit Back to Main Menu 

Enter Number of Your Choice: _ 
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Percent Passing Grade 3 Objectives 
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Profile Development and Goal Setting For 
School Improvement Leadership Teams 

Bob Blum 

Northwest Regional Educational Laboratory 

I appreciate the opportunity to share a few ideas about what we've 
been doing with school improvement, particularly what we call profiling. 
I will describe two parts of what we do: developing a profile of student 
performance and setting improvement goals. 

I always like to get my biases out early in a presentation. I heard 
a bias coming out at the end of the last presentation, something about 
the relative importance of State vs. others, and so jn. Bias number one 
is that we are in the business of schooling to get all students to learn 
well. That 1 s a very basic bottom line belief and I underscore all 
students and learning well. I'll define these concepts more specifically 
as we go along. 

The sect .id bias is that the primary purpose of student information, 
or student performance data, is to drive improvement. There are many 
other purposes: some additional purposes are being accountable and 
reporting to parents, but the primary purpose cf student performance data 
is to drive improvement. A third bias is that improvement does not just 
happen. Someone must manage it. Someone has to initiate it, care for 
it, feed it, walch it, pat people on <*he back for doing it, and whatever 
else is necessary. Improvement has to be managed. It is the student 
performance data that provides the good information for the managers of 
improvement processes. In our work, the managers of improvement are 
called a leadership team- The team is a group composed of principal, 
selected teachers and someone from the central office. It nicely 
combines three levels: district, school, and classroom, to manage 
improvement in a single school. 

A fourth bias is, and research supports this, that the school is the 
unit that has the best chance of getting improvement done. The classroom 
level is important, teachers have direct contact with students, and they 
can and do improve one at a time. The district is important because it 
can encourage, support and mandate improvement. The state is important 
because states establish requirements, encourage or require improvement, 
and provide resources. But the right level to get improvement done is 
the school. Schools have the right combination of people; have common 
interests — a mission — and have has access to the students. The school is 
where the action is. 

A fifth bias is that the data base, a profile of student performance, 
should be broad rather than narrow. There should be many indicators 
rather than just one or two. It should not be focused only on basic 
skills achievement. We must look for indicators of many aspects of 
student performance The advice we give schools as they create their own 
data buses is to collect information in st least thr»e ar >as of student 
performance: academic achievement, social behavior and attitude. And 
when thinking about academic achievement, they should think broa. y. Use 
standardised tests. Use information from teacher-made tests if they are 
well constructed and information can be collected systematically. 
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Collect information about reading, writing, and math; and collect 
information about social studies, science, art, physical education, and 
more. Collect information on higher level thinking and study skills. 
Think about collecting information on student performance in all skills. 

When advis'ig schools aboot creating a profile of student 
performance, we suggest that they think about a couple of terms that are 
used frequently, but not well defined: effectiveness and excellence. 
Effectiveness typically means that all of the studcmts master basic 
priority objectives. The basic priority objectives usually mean that 
students do well on standardized criteria or norm-referenced tests of 
reading and math achievement. Excellence means that most students learn 
much more than the basic priority objectives in reading, math, and 
language arts. He suggest that schools think about higher level thinking 
skills and study skills. Think about students doing well in advanced 
placement courses and in all other subjects. Think about a whole ranee 
of indicators that go well beyond the very basic skills. Think about 
effectiveness and think about excellence. Decide what these terms m /.i 
to you in your school and/or school district. Think about the data jou 
need to determine whether or not the students are learning well in basic 
areas and well beyond. 

Th~ sixth and final bias is that the easy part of profiling, or data 
base mangement, is getting the data together. The hard part is getting 
it used for the purposes intended: to drive school improvement. That's 
another bias. 

Now, I want to talk about the document called a profile. It is the 
written description of student performance for a school. Schools develop 
their own profiles. They are all difierent~but all schools try it and 
they all get a profile developed. The quality of profile is from a very 
good to poor. The advice we give is the leadership team has the 
responsibility to develop the profile. Start with the data you have, and 
then add important types of information. We give schools a "profile 
contents menu" that includes indicators of student performance in *he 
three areas I mentioned earlier, academic achievement, social behavior 
and attitude. We suggest that the leadership team go through the menu 
and check the kinds of information that they already have and decide how 
tuey will pull this information together. The teams then decide what 
additional information they want and how they will collect it. They have 
to decide what existing and new data will go into their profile. 

We also advise schools on the character of the data. We suggest that 
what they need is school level data — school level, not classroom by 
classroom, not grade by grade, but school vide. School level data are 
needed to set schoolwide 4 improvement goals. Everyone in the school is 
going to contribute to achieving the school improvement goal, so you want 
to see how well the school as a whole is doing. As improvement 
progresses, teachers, grades and departments need to see how well they're 
doing, but for goal setting, it is the school level data that is needed. 

We su v y est that the data be .longitudinal. Get multi-year data when 
putting together information on standardized test scores. Look at 
indicators such as the percentage of kids in the top and bottom guar tiles 
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over a multi-year period. The data should be aggregated to the school 
level, so you will see several years of data on percentage of students in 
each quartile of your school in the basic skills areas. 

He also suggest that schools use the biggest numbers from the measure 
they use — total math, total language arts, total whatever. Use the big 
numbevs. You want to create a picture of your whole school, using a few 
key indicators for how well the school is performing. 

It is also important to disaggregate student performance data 
according to some criteria like socio-economic status of family, sex, or 
race. For example, you may want to look at percent in the lowest 
quartile and the highest quartile for the students who are on free and 
reduced lunch and for those who are not on free and reduced lunch. You 
may also want to look at similar data for boys and for girls. Some 
schools have had interesting findings. For instance, one school found 
that girls do consistently less well on most measures (SAT, grades in 
college, etc.) than boys do. The school can choose to ignore the 
finding, but they at least know the situation. Disaggregating data is 
important* 

He also suggest that schools show district level, and perhaps state 
and national level data, along with their own. This provides a point of 
reference for schools to use in evaluating their own performance. 

We advise showing data graphically as well as in chart form. Make it 
simple because all faculty members, and perhaps the community, will look 
it over. We also suggest that schools study the information carefully, 
and pull out the key facts that describe student performance, e.g., 
during the past three years the percentage of students in the top 
quartile has declined from 18-14 percent. These factual statements are 
called narratives and are included in the profile. The narrative must be 
simple and clear so that staff and others understand the facts and can 
make decisions based on the facts. The Leadership team writes several 
narratives based on the data they have, and goals are derived from the 
narratives. 

I've been talking about getting the profile together. I call that 
the easy pert, even though it takes a lot of time and energy. Different 
teams do it different ways. They split up the data and have various 
people work on parts. They have central office staff do some of the 
work. They scurry around and find all kinds of information, and they do 
a great job of developing their first profile. When the profile is 
completed, it belongs to the school, and they are fascinated by what they 
see and don't see within the informaton. But that's the easy part. 

In most cases the hard part is getting the information used right. 
We've suggested a very structured approach to using the data to move from 
a broad set of indicators of school performance to one or two goals for 
improvement, one or two goals that all faculty agree upon. Getting 
agreement and * buy-in* is the important part . Goal setting brings people 
together on important improvements that they want to make. 



The formalized process we suggest is one in which leadership teams 
start with their profile that includes data displays and narratives in 
the three areas of academic achievement, social behavior and attitude. 
They give the full profile to the faculty for review and analysis. The 
team pulls out the most important narratives and puts them onto a form so 
that the staff can analyze and evaluate current schoolwide results. They 
ask the group how they feel about the current results. Are you satisfied 
or dissatisfied with the various results as described in the narratives? 
How do you feel about the fact that the percentage of students in the top 
quartile decreased over five years? Is it important or not important? 
Are you satisfied or dissatisfied with this result? 

The next question in the evaluation prjcess relates to the relative 
importance of various results. Are the wath test results more important 
than the attendance results to you at fiis time? Two questions: how 
satisfied, how important. The full faculty systematically analyzes and 
evaluates the narratives. They do this individually and in small 
groups. The leadership team takes the evaluation from the whole faculty, 
tl*e concensus ratings that come out of small groups, and transforms 
factual narrative statements into preliminary goals. A preliminary goal 
might be to improve math achievement. If the faculty is very 
dissatisfied with results in math and think it's important, the 
leadership team transforms this into a preliminary goal. Take several 
preliminary goals back to the faculty for one more round of input. The 
outcome is one or two goals for improvement. Everyone has had a chance 
to be involved. The time has been made available, the group process 
happens and the school settles on one or two goals for improvement. 

And that's the hard part—getting the data used to establish one or 
two goals for improvement in student performance. We suggest a very 
formal process. I have a hunch that when schools go through the process 
for the second or third time, they will modify and streamline it. But 
schools do need some formal process for engaging the whole faculty in 
analysis of current results and goal setting. 

The Goal Based Education Program is primarily interested in school 
improvement. In the early going, we are willing to take a little less 
technical quality in collecting and displaying student performance data. 
Almost all the profiles are imperfect at first, although some of the 
schools now are doing a very, very good job. After a school goes through 
this once, they begin to improve their profile. They realize that their 
data is really not quite as good as it ought to be. They decide they 
need additional information. They realize that a particular graph is^t 
telling the story, or that something f s too complicated. They initiate a 
process of improving the profile at the same time that they are improving 
their school. The school is in control. They are striving for school 
improvement. It is their data and they improve it over time. 

To sum up, we believe that the school is the right level for 
improvement. The school neeis a broad database on student performance, 
not just basic skills achievement. And it is critically important for 
the school to use their profile to engage the whole faculty in setting 
goals for improvement. 
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District Use of Data to Support School Improvement 

Michael Brott 
Central School District, Oregon 

My primary focus is on monitoring student progress and on an 
evaluation system for this purpose, Ao a district we are working on the 
other areas of effective schools, particularly a coordinated curriculum, 
but for today's purposes we are not talking about those other areas of 
effective schools, aven though they're very important. 

We have a firm belief, that the teacher is the center of the 
educational process. The teacher is a key decision maker within our goal 
directed/outcome based in&tructural program. The coordinated curriculum 
that is used is designed with high expectations for specific learning 
goals and objectives. The teacher has many decisions to make including 
developing the instructional plans and procedures to be used to f osier 
goal attainment, plus assessment plans and procedures that monitor 
progress toward goal attainment. 

Central School District is a member of the Valley Education 
Consortium, a consortium of small school districts in Marion, Polk and 
Yamhill Counties, working with Teaching Research from the Department of 
Higher Education, and OSU/WOSC, Through this consortium we have been 
able to develop goal directed/outcome based curriculum, with the 
corresponding assessment systems, A teacher can use the test item pool, 
which is a thick document containing test items in math, reading, 
science, or whatever is needed. Teachers can pull from the test item 
pool at any time to measure a particular goal or set of objectives. 
Feedback can be provided to students, parents, and the school 
administrator on how those students are progressing towards particular 
goals at any point in the student's learning, 

I won't go through the rest of this diagram, but I'll be referring 
back to it at seme point, I simply wanted you to know that this is the 
context from which the data we working with really comes. 



Three Levels of Monitoring 

We talk to our administrators about having three levels of 
monitoring. The first level is the one we all know about, the teacher 
level . We expect our teachers to monitor on a lesson-by-lesson basis, as 
all good teachers do. They'll ask questions, ask students to demonstrate 
or signal their understanding. These are things all of us who observe 
teachers see on a regular basis* Next, we know that teachers are going 
to be giving unit tests. They will use teacher-made unit tests, or 
textbook tests, or whatever they have available, to measure how well the 
students are progressing in a given unit of study. The third part of the 
teacher monitoring is the use of test item pool. This component of the 
teacher monitoring is not common. The teacher will use the test item 
pool to measure the skills and concepts that have been outlined in the 
goal based curriculum. The use of the test item pool is e key factor in 
determining* progress towards mastery of the curriculum. 



The principal is another important person in this process for school 
improvement. We expect our principals to be able to walk into the 
classroom on a formal or informal observation and be able to monitor the 
teacher in the process ot monitor* ~g an individual lesson, see that the 
teacher is using the instructional techniques that we know are effective, 
and to be able to gather data by asking questions. For example, if the 
principal sees a group of students working on literal comprehension in 
the back of the room, he will be able to come back to the teacher and 
sayt "I notice that these children were working on literal comprehension. 
Can you tell me where that f in your scope and sequence, where you are 
at this time?" Hopefully, the teacher can give that information and 
share with the principal where they are in the scope and sequence. This 
does two things; (1) It puts the principal in the position of setting an 
expectation for that particular teacher, that u l am concerned that you 
are teaching the district curriculum and you are moving towards the goals 
that we have agreed upon." (2) The principal generally will ask, "How 
well are the students in this group doing towards that goal?" The 
teacher has to be able to give specific information on the particular 
criterion referenced tests that they have used to show how those students 
have demonstrated mastery (or nonmastery) of that skill. It is very 
important for our principals, as well as, the teachers to know that this 
is happening at individual classroom level. 

The second level is the building administrator level. We believe 
that each building should have individual student data, and the principal 
is th* key to this occur ing. We expect the principal to have individual 
student data readily available, not only for use by the principal, but 
for use by the individual teacher. We can ue* such information as 
[overhead of a test card is put on screen, see Appendix F] • This is a 
test we've used. We are about a year away from having all this data on 
the computer where we can have ready access to it. This is a fourth 
grader who came to us in the Fall and we have & Fall and Spring score 
shown on the top and bottom for fourth and fifth grades. We can begin to 
follow the progress of that particular student and see growth trends. 
The teachers can pull the cards on a particular group of children and see 
the profile of the members of that group and plan accordingly. 

A second component of the iron it or ing system is what I call the wat ch 
list, and this is where we get some of our better improvement . We expect 
each of our buildings to identify the point at which students are going 
to be given additional assistance. A standard must be established for 
the data in order to raise a red flag that tells us we need to do /one 
further work with a particular student. To give you an example, 
[referring to overhead transparency, shown at the *nd of this paper] if 
we take a set of 3rd grade scores ordered by reading, and looking down 
this list of reading scores, we can identify these tvo students who fall 
below the 30th percentile. This would be the first step in raising the 
red flag on a couple of students that may be put on the watch list . The 
school would then look at those students 1 individual test cards to see if 
this pattern was something that had occurred over time, or was just a 
one-time occurrence. The school would want to look at the criterion 
referenced testa and see what kind of scores the students had on the goal 
based tests to determine whether or not they had mastered the appropriate 
goals in reading and math. If the students were at the 50th percentile, 
less than the expected 80 percent mastery level, that would automatically 
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This is a way for us to identify those students who have some 
academic difficulty. Once the studentr are identified, it becomes 
necessary to find out what those children know and what they don't know, 
on a specific basis. The school needs to know how well the students are 
doing in reading and it must use a finely tuned diagnostic system, fcnen 
it is determined what the students hnow and don't know, the school 
proceeds to program planning. At the elementary or middle school, it is 
assumed that a student needs more than their regular program. Obviously, 
if the child is not at the level they should be, the school has to do 
something more than is provided in the regular program. The regular 
program is examined to make sure it fits what the child needs and then 
another program is designated for providing a double dose for that child, 
or a triple dose if necessary. The double dose program may be a 
Chapter 1 program in Jie school, if the child qualifies, or it may be a 
special ed mildly handicapped classroom, if that is indicated, or the 
school may r»eed to simply put the child into a tutor program so that at 
some time during the day they are able to provide for an additional 
instructional period. 

Frequent monitoring is the key. If the school is going to double 
dose that child, then every six to nine wc^ks it needs to do a quick 
assessment in those particular areas where the child was weak, because if 
there isn f t twice the growth that would expected under normal 
circumstances, then the program isn't working. If the school is not 
getting that kind of growth then something is wrong and corrective action 
must be taken. 

Those >re the pieces we think need to be in place for the ' ailding 
principal monitoring program to be sure that kids aren't falling through 
the cracks. He monitor to make sure the program that we expect to be 
trught, is being taught, an4 that each child is making the progress that 
we think the child is capable of making. 

The third level of the building administrator's monitoring system has 
to be classroom information. The principal needs standardized test data 
and criterion reference test data. Let me }ust share with you briefly 
what some of our classroom data looks like. [Overhead put up, see 
Appendix.] 

Here is a set of fifth grade mathematics for our criterion referenced 
test. This is a comparison between mid-winter and spring and we 1 re 
primarily interested in the gain scores. The essence of this data 
indicates that different teachers achieve different different results 
with each of the quartiles. Some teachers seem to work better with lower 
quartile students, while others work well with all students. A principal 
needs this data as a major part of his building monitoring system. 

The third, and final, component of our monitoring system is the 
district level . The superintendent must have a comprehensive data 
package that includes standardized and criterion reference data for the 
district, school, and grade level. 

The tact results must be reported to the school board and public and 
form the foundation for program evaluation. The criterion reference 
tests provide the necessary data for determining the degree of m~~*-«ry 
for the district required curriculum. (See Appendix.) 
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SS • Standard Scora 

% • Pare tn tile 

tt.E.* Grada Equivalent 



BEST COPY AV/toUfi 



GRAD E 3 RECORD 
METROPOLITAN TEST 



DATE ^>/tJ t 




FIFTH GRADE MATHEMATICS 
DISTRICT A 



Mark Schalock 5/6/85 
For Discuss ion On I y 



Percent Average % * Average NCE Average 

m-TO,-, . Mid-Year End-of-Year Grin of Potential Standardized Student 

UI5IK>U A ... Scor e Score Score Gain Realized Test Score Attitudes 



>tal District 


(N « 


128) 


53-59* 


63.13* 


9.54* 


21.S<i* 


56.68 


Top Quart Ue 


(n - 


3D 


70.27* 


82.06* 


11.79* 


** 

38.42* 


77.03 


Second 


(n - 


36) 


56. 19? 


62.83* 


6.64* 


15.18* 


53-78 


Third 


(n - 


30) 


48.86% 


59.33* 


10.47* 


21.35* 


52.07 


Bo.com 


(n » 


3D 


38. 47* 


48.61* 


10.14* 


15.33* 


43.74 



3-5J 



4.03 
3.8J 
3.20 
2.37 

Teacher attitudes toward the V£C Mathematics program - 38.92/60 
£ Teacher Implementation of the VEC Mathematics program - 20.05/ 36 

* » * , (E nd-of-Year) - (Hid- Yea r ) 

* Average * of potential gain realized - ~~ 

100* -(Mid-Year) 

** ^ e, «J ,s f p,bu t | on» represent patterns that ere significantly different from what you would expect If the 

1 $ L fl? n , 0f ?* ,0S Were equa,,y dl$ f ,b "t«d across ell students. This Is based on the Chl-Square Test; 
at the 0,05 levei. 
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FIFTH GRADE MATHEMATICS 
SCHOOL 1 



CLASSROOH 1-A 


IN » 




Mid-Year 
Score 


End-of-Year 
ieore 


Percent 
Gain 
Score 


Average * 
of Potential 
Gain Realized 


Averaae NCF 

Standardized 
Test Score 


Av/Af ar<a> 

pwcroyc 

Student 


Trt f* A 1 riser waam 




60. 16? 


71.77* 


11.61* 


27.41* 




^ui vuoes 


Top Quart He 


(n - 


4) 


74.63* 


79.35* 


4.73* 


16.98* 


69.75 


4.50 


Second 


(n - 


5) 


65. 42* 


74.94* 


9.52* 


28.18* 


70.00 


3.80 


Third 


(n - 


4) 


55.35* 


69.65* 


14.33* 


31.28* 


64.75 


2.25 


Bottom 


(n - 


4) 


43. 93* 


62.33* 


18.40* 


33.03* 


54.25 


3.50 



Teacher Attitudes toward the VEC Mathematics Program - 35 
Teacher Implementation of the VEC Mathematics Program - 16 



CLASSROOM J-B 



Total Classroom 


(N ■ 


15) 


52.37* 


51.18* 


-1.19* 


0.62* 






Top Quart He 


(n » 


4) 


65.73* 


75.33% 


9.60* 


28.80* 


82.00 


4.00 


Second 


(n - 


4) 


57.13* 


52.68* 


-4.45* 


-10,00* 


52.25 


4.50 


Third 


(n ■ 


3) 


45.23* 


40.80* 


-4.43* 


-7.23* 


45.00 


5.00 


Bottom 


(n - 


4) 


39.63* 


3?. 33* 


-6.30* 


-11.05* 


37.74 


4.00 
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[JiSy^ Teacher AttJ codes. toward the ^ Mathematics Program - 40 
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FIFTH GRADE MATHEMATICS 
SCHOOL 2 



Mid-Year 



End-of-YEar 



Percent 
Gain 



Average X 
of Potential 



Average NCE 
Standardized 



Average 
Student 



CLASSROOM 2-A 




Score 


Score 


Score 


Gain Realized 


Test Score 


Att i tudes 


Total Classroom 


(N « 15) 


62.951 


68.45* 


5.50 


15.72* 


62.47 


3.40 


Top Quart i ie 


(n = 4) 


76.1.3* 


86.00* 


9.57* 


39.83* 


88.00 


3.25 


Second 


(n = 4) 


64. 30* 


62.65* 


-1.65* 


-3.95* 


55.25 


3.25 


Third 


(n « 3) 


61 . %3* 


70.67* 


9.24* 


23.87* 


51.33 


3.33 


Bottom 


(n - k) 


49.28* 


55.03* 


5.75* 




a en 
>Z.?U 


u oo 

•f • WW 


Teacher attitude towards the VEC Mathematics program 
Teacher implementation of the VEC Mathematics program 


- 50 
- 19 








CLASSROOM 2-8 
















Total Classroom 


(N - 16) 


53.30* 


66.76* 


13. 46* 


31.23* 


61 .50 


3.31 


Top Quart 1 le 


(n - 4) 


66.05* 


82.00* 


15.95* 


49,08* 


80.25 


3.25 


Second 


in * 4) 


58.20* 


68.68* 


10.48* 


24.93* 


61.75 


3.50 


Third 


(n - 4) 


52.88* 


66.70* 


13.82* 


29.73* 


55.50 


3.50 


Bottom 


(n ■ 4) 


36.08* 


49.68* 


13.60* 


21.18* 


48.50 


3.00 



Si 



Teacher attitude towards the VEC Mathematics progran. » 38 
. icr Implwwntitlon of the VEC Mathematics pre un - 19 

SERLC 72 
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FIFTH GRADE MATHEMATICS 
SCHOOL 3 



Average Average * Average NCE Average 

Mid-Year End-of-Year Gain of Potential Standardized Student 

CLASSROOM 3-A Score Score Score Gain Realized Test Score Attitudes 



Total Classroom (N - Ik) 50.6lt 70.24* 19.63* 40.76* 60.93 3.07 



Top Quart He (n - 3) 70.47* 85.33* 14.86* 49.47* 77.00 !♦ . 67 

Second (n » 4) 51.43* 69. 08* 17.65* 36.50* 56.75 3. 50 

Thlr <* (n - *) 48.60* 69.33* 20.73* 40.30* 58.50 2.75 

Bottom (n - 3) 32.37* 57-90* 25.53* 38.23* 53.67 1.33 

Teacher attitude towards the VEC Mathematics program - 27 
Teacher Implementation of the VEC Mjthematlcs program - 26 

CLASSROOM 3-B 

Total Classroom (N - 13) 51.55* 63.86* 12.85* 27.85* 54.62 3.08 



TopQuartlle (n - 3) 66.20* 8$. 33* 19.13* 55. 80S 82.00 U3 

Second (n«4) 52.18* 58.03* 5.8ft 12.10* 45.75 3,00 

Third {n - 3) 48.57* 67.33* 18.76* 36.33* 53.00 2.67 

Bottoc (n • 3) 39.07* W,7Q* 7.63* 12,1)6* 40.67 2.33 

174 175 
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A GOAL-BASED INSTRUCTIONAL PROGRAM 

MAKtiS 
A 

DIFFERENCE 



READING TOTAL 



District Mean 
District Mean 



55th Percentile 
65th Percentile 



Growth 10 Percentile Points 



LANGUAGE TOTAL 



District Mean 
District Mean 



54th Percentile 
65th Percentile 



Growth 11 Percentile Points 



SPELLING TOTAL 



District Mean 
District Mean 



54th Percentile 
61st Percentile 



Growth 7 Percentile Points 



MATH TOTAL 



District Mean 
District Mean 



53rd Percentile 
59th Percentile 



Growth 6 Percentile Points 
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TABLE III 
Summary of District CAT Scores 



MATH PERCENTILES 
Math Computation 



Grade 



Year 


2 


3 


4 


5 


6 


7 


8 


9 


10 


1981-82 


— 


59 


49 


52 




50 


51 


51 


52 


1982-83 


— 


61 


50 


50 


62 


67 


59 


58 


57 


1983-84 




46 


56 


56 


57 


62 


57 


58 


61 


1984-85 


43 


52 


54 


61 


55 


56 


56 


CO 

DO 

• 


cn 








Math Concepts and Application 






Year 


2 


3 


4 


5 


Grade 
6 


7 


Q 

O 


9 


10 


1981-82 




52 


51 


53 






DC 


51 


68 


1982-83 




57 


55 


55 


55 


60 


56 


59 


57 


1983-84 





53 


64 


54 


60 


61 


61 


61 


60 


1984-85 


54 


62 


65 


71 


56 


61 


64 


65 


62 










Math Total 










Year 


2 


3 


4 


5 


Grade 
6 


7 


8 


9 


10 


1981-82 




51 


50 


53 






52 


51 


61 


19^2-83 
1983-84 


— j 


61 

50 


52 

63 


50 
55 


58 

60 


64 % 

63 


59 
59 


60 
60 


58 
60 


1984-85 


49 


57 


60 


66 


57 


59 


61 


62 


61 










Battery Total 










Year 


2 


3 


4 


5 


Grade 
6 


7 


8 


9 


10 


1983-84 




55 


66 


60 


64 


66 


62 


61 


62 



TABLE I 

Summary of Ol$t» let CAT Scores 
READING PERCENTILES 



Rg&jjSg Vocabulary 



Year 


2 


3 


4 


5 


Grade 
6 


7 


8 


9 


10 






52 


52 


S5 




54 


53 


12 


7? 


1982-83 




58 


53 


54 


56 


64 


61 


60 


61 


1983-84 




61 


64 


62 


63 


69 


67 


64 


63 


1984-65 


61 


61 


61 


65 


60 


64 


64 


64 


61 



Reading Comprehension 



Year 


2 


3 


4 . 


- JL 


Grade 
6 


7 


8 


9 


10 


1981-82 




54 


51 


56 




53 


53 


53 


60 


1982-83 




53 


5o 


57 


56 


66 


56 


5G 


61 


1983-84 




58 


65 


60 


65 


69 


64 


64 


65 


1984-85 




64 


65 


67 


59 


64 


69 


70 


64 


i 


i 




• 


Readina Total 
• 


% 








Ytsr 


2 


3 


4 


5 


Grade 
6 


7 


8 


9 


10 


198 i-8? 




53 


52 


56 




53 


53 


53 


66 


1982-63 




58 


55 


57 


56 


66 


59 


61 


61 


1983-84 




64 


65 


52 


64 


69 


66 


65 




1984-85 


65 


66 


65 


68 


50 


65 


67 


69 


63 
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Grade Pour 

STUDENT MASTERY OP LEARNING GOALS WITHIN GRADE LEVELS 



Goal 




Numeration 

Mastery 
Parl-rrastery 
Non-njastery 
Whole Numbers 
Mastery 
Part-mastery 
Non-mastery 
Complex Wora P obtems 
Mastery 
Part-mastery 
Non-mastt^y 
Measir - . *nt 
Master 
Part-mastery 
Non-mastery 
Geohb^try 
Mastery 
Part-mastery 
Non-mastery 

Fractions 

Mastery 
Hart-mastery 
Non-mastery 



61. OS 




33.5* 



148 180 



s 
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VEC MATH TESTS 

Comply: Word Probi^m-a 
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District Use of Data To Support 
Basic Skill? Program Improvement 



Bob Hammond 
Springfield Public Schools 



Springfield is a community of 40,000 people. We have approximately 
9,000 students in fifteen elementary schools, four middle schools and two 
high schools 

For the past seven years, the district has been at work developing 
ways to examine its basic skills instructional programs. One of several 
procedures developed allows us to monitor student performance from year 
to year using our standardized testing program. 

In our di8tri'"t, standardized tests are administered in the fall and 
where needed, are followed up with individual diagnostic testing. 
Currently we have scores for grades 1 through 12 on the California 
Achievement Te?t. 

From the assessment data provided by our standardized testing 
program, how do k? locate problem areas to improve programs: We know 
from examining our district's average scores that our population's 
performance is generally similar to the national population on which the 
California Achievement Test was normod. Given this similarity, how far 
can our average scores vary from the national averages before we wust 
conclude that there is a need for local program improvement? Also, how 
far can our average 3cores vary from one year to the next be for h we 
conclude that some aspect of a program is responsible for sporadic ups 
and dotfns? 

District standards have been set which allow us to determine how far 
is "too far." The standards are based on standard acores and the normal 
curve concept—but before looking at these standards, let 1 * take a look 
at the rationale behind them. 

Judgments about performance are not made in a vacuum. Only by 
comparing two or more sets of data can we say anything about a group's 
relative achievement. 

The most basic measure provided by any test is the ra;; ?core. On a 
test for third grade reading, there are 125 items, thus a gorup could 
theoretically obtain a raw score average from 0 to 125. The national 
average on this test iz 80. For grade 4 reading, there arc 127 items and 
scores could vary from 0 to 127. The national average on this test is 
70. Obvioucly, a score of 70 in one Jts^r is not the same as a score of 
70 in the next. 

How can we compare the third grade's performance in one year with 
their next year's fourth grade performance? Also, what more can we 
conclude about the relationship between the district and the nation other 
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than noting a vague way that a score is above or below the national 
average? With raw scores, what do we have to compare? 

Raw scores can be converted into another score which will allow us to 
compare achievement ever time. With scale scores, we can compare 
achievement from one grade to the next because these scores are based on 
a continuous scale. Scale score were developed by the publishers of CAT 
in such a way that the national average scores increase from grade to 
grade. 

Another scale exists which will allow us both to compare scores from 
one year to the nest and to compare *he district with the nation. This 
is the standard score scale, which is based on the concept of the normal 
curvtt. We can use this scale since we know that our district's 
performance is similar to the national norm gtoup and since scores of the 
national norm group *it the normal curve pattern, with standard scores, 
the average is always set at zero and a given percent of student scores 
will fall within specified ranges of deviation from the average, witha 
majority of students scoring close to the average. 

On the standard score scale, the ranges are marked by intervals of 
one. A standard score reflects the degree of deviation from the national 
average. Two-thirds of the natio* . population scored within one 
standard deviation above and one .umdard deviation below ihe average. 
Another way of saying this is that two-thirds of the national population 
had a astandard score between plus or minus ona. The boundaries set for 
our district are one- fourth or a .25 of a standard score above and below 
the natonal average. Scores falling within those boundaries will be 
considered within the average variation for our district compared with 
the nation. If a district average falls below the boundary „ it will be 
considered below the expected performance for our district. 

Besides allowing ns to determine where a single score is in relation 
to the nation, the standard score and our boundaries let us compare the 
district average in a skill area from one grade and year to the next to 
see if a program r .mains within its own expected range of variation. 
Depending on both the ccrrent stanrdard score average and the grade level 
difference score* there are 4 possible sets of judgments that can be made 
(see decision rule chart) • 

In review, the assessuent process has given us data for locating 
problem program areas. By applying district standards to the data, : is 
recommended that a needs identification and program improvement be 
conducted. The State Minimum Standards requires that needs be identified 
in the areas assessed, that priorities he set -.or meecing those needs and 
that policies and procedures be designed for making program improvements. 

The assessment process allowed us to locate instructional areas which 
are not meeting district standards. The next step is to identify unmet 
program needs which one can infer are having a negative impact on student 
performance in these areas. 
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PROGRAM IMPROVEMENT DECISION RULE 



OEC ,d AT OR 

RULE 1 ABOVE STANDARD 



DISTRICT STANDARD SCORE 
FALLS BETWEEN A -.25 AND 
because +.25 OR ABOVE +.25 WHEN 
COMPARED TO THE 
NATIONAL AVERAGE 



DECISION 
RULE 2 



BELOW STANDARD because 



THE DIFFERENCE BETWEEN 
THE CURRENT ANDTHE 
PREVIOUS INSTRUCTIONAL 
YEAR IS -.25 OR GREATER 



DECISION 
RULE 3 



BE LOW STANDARD because 



DISTRICT STANDARD SCOR* 
AVERAGE FALLS AT OR 
BEtOW-25 



DECISION 
MULE 4 
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THE DIFFERENCE BETWEEN THE 
CURRENT AND THE PREVIOUS 
.HARD BfCAos '\'STm«CTlONAL VEAR IS . 25 

• 'ft GRLAFER AND THE CURRENT 

district standa rd score 
fall5;bflovv ..25 



1983 



1984 




PROGRAM IMPROV EMFNT 

iSMft T required: 

STAFF SHOULD BE AWARE 
CH "WHAT IS NEW IN THE 
LITERATURE". 



1983 



1984 




PRO GRAM IMPROVEMENT 
IS NOT F*QUIREP7 
PERFORMANCE HAS DROPPED 
SIGNIFICANTLY. 
TEACHERS, PRINCIPALS, & 
SPECIALISTS SHOULD BE 
ALERTED FOR 
fOTENTIAL PROBLEMS. 



1983 



1984 




PROGRAM IMPROVEM ENT 
IS RECOMMENDED 

PROBLEMS MUST BE 
IDENTIFIED AND ALTERNATIVE 
SOLUTIONS EXAMINED FOR 
POTENTIAL IMPLEMENTATION 
ACCORDING TO DISTRICT 
DEVELOPED PRIORITIES. 



1983 



1984 




P ROGRAM IM PROVEMENT 
ISRfeftUtftEft 

PROBLEMS MUST BE 
IDENTIFIED, ALTERNATIVE 
SOLUTIONS EXAMINEC *• 
ACTIONS TAKEN TO 
CORRECT PROHLf M AS 
SOON AS POSSIBLE. 
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District Use of Data 
To Support School Improvement 



Dr. Milton Snyder 
Federal Way School District, Washington 



THE FEDERAL WAY STORY 



Federal Way has 15,000 students with 700 teachers and 750 support 
personnel. Federal Way School District is located in a large suburban 
area between Seattle and Tacoma and 15 minutes from Sea-Tac International 
Airport. 

Ife have gone through a devastating strike in the 1970*8, lost levies 
for seven successive years; saw test scores and student achievement 
decline because of lack of human and material resources; and the loss of 
community and staff faith in the District's ability to educate children. 

Today, the District boasts 2/3*8 of its student test scores are above 
graoe level on National tests, we are showing an enrollment increase for 
the first time in 10 years, since 1980 we have passed three two-year 
levies and a $30 million bond issue* Our bus fleet is being modernized 
with the purcM** of 22 bus^s over the last few years and the surplusing 
of *any older uses; the building and grounds look like we care once 
again; we haw* either PTA's or Booster Clubs in every school; and 2/3 f s 
of our 26 school 8 have been accredited with a goal of 100% accreditation 
by the end of next year. Pride and dedication have replace apathy. 

HOW DID WE START? 

Goal setting was a must. Zn 1980, the District gathered in~ut from 
community members on their perceptions of the schools, staff and 
curriculum. Goals were set to address the most critical preceived 
needs. Unaccomplished goals were carriad forward as new goals were set. 

I am an advocate of community participat a in the schools whenever 
possible. I have a counitment to ensure parent and ethnic representation 
at all levels. Our first People-to-People meeting was held in 1980. For 
two years a series of meetings were* held on specific topics with members 
of the community invited to attend. While this was well received, we 
still did not draw the numbers I felt were needed. 

Through Board review, community input, staff and cabinet work, we 
were able to set academic program goals as well as goals in the areas of 
in-service for certificated and classified staff, building and grounds 
improvement, financial accountability, etc. 
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I developed my own priorities with cabinet. Each division — Learning 
and Support — developed their goals based upon community , Board and my 
goals. Learning goals were developed with the assistance of the 
principals. They were then filtered down to the key element— the 
school — where individual building plans are developed. After all, the 
place where the goal must go into action is the classroom, and the 
results are with the children. 

LET'S LOOK AT THE PARS THAT MAKE UP THE PLAN 

The community input is not developed in a haphazard manner but 
through a planned, systematic approach. You cannot decide on a Friday to 
put Community Forum on the following week. 

Pnase I includes a survey distributed in the beginning of October. 
Until this year, they were distributed to all staff, Key community 
leaders, VTL and Booster Club officers and identified individuals. This 
has changed this year as we have developed a survey which will be mailed 
to 30,000 postal patrons within our school boundaries in addition to the 
groups and individuals previously mentioned. 

The one negative point in previous years is that we dealt with 
positive groups— —i.e. parents, community activists with kids in the 
schools and for the most part, a positive staff. We will alleviate this 
negative point Uiis yeai through the Survey developed by our 
Communications Person. 

In Federal Way we have five junior high scnools which serve as the 
sites for the Comaunity Forums , Each of the 26 schools guarantee 10 
people who will attend one meeting in their service area. The forums 
generally lt*st approximately 3 hours each and include a brief address by 
a Board mejiber, an overview of the District and identification and 
discussion of 10 critical issues (obtained through the results of the 
first survey), time for small group discussions, completion of a 
different survey, and then a general question and answer period. 

The^e results are then made available to the Board who -*tvelops the 
District goals while participating in a one-day workstudy. By this 
point, they have received the community forum results, information from 
the Superintendent, Board and Cabinet, and from the perceptions of needs 
and desires they bring with them that day. 

Once they have developed their goals- I begin to work on mine — based 
on the District goals. 

The next step is the development of administrative goals by each of 
the divisions. These are the goals that give the foundation and state 
who will do what by when. Their plan of action is then developed — just 
as School Learning Plans which support all other levels and give action 
and direction. These goals must have a plan of action and include who 
will carry out the goal, by when, how and what type of evaluation will be 
used to see if the goal has been met. The key ingredient is the 
individual Learning Plan in each of the District's 26 schools. 
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A Learning Plan is nothing more than a partnership betwean the 
principal, professional and certificated staff and the community. 

The Learning Plan committee is composed of principals, 
representatives of every elementary grade level, broaa subject areas at 
the secondary level and community members. All Learning Plan goals 
support the need for achievement and improvement and also support the 
Board's goals. 

Specifically, the goal is stated in measurable terms—once again who 
is responsible for achieving the goal, how will the goal be accomplished, 
when and how it will be evaluated. 
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edcral Way School District 




SCHOOL 

WARNING- 
PLAN 

19.84-85 
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RESULTS 
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MIRROR LAKE ELEMENTARY 
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LOOKS AT THE BIG 
PROBLEM, TAKES IT 
APART, AND SOLVES 
THE SMALL PARTS! 

Ill Hill 

TAKES A LOT OF THE 
LITTLE PROBLEMS & 
ROLLS THEM UP 
UNTIL THEY ARE 
UNSOLVEABLE! 
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* SUPPORTING 
DISTRICT GOALS 

* REFLECTING 
SCHOOL NEEDS 

* FOCUSING ON 
LEARNING 
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ANALYZE THE TEST DATA 

* IDENTIFY STRENGTHS 

* IDENTIFY WEAKNESSES 

ANALYZE WEAKNESSES 

* PAST PERFORMANCE 

* LONGITUDINALLY 

* COMPARISONS 

* ITEM ANALYSIS 

* SLO MATCH 

* CURRICULUM MATCH 

* SEQUENCE MATCH 
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DETERMINE MEEDS 

* TEST DATA 

* IMPUT FROM: 

* STAFF 

* STUDENTS 

* PTA (PARENTS) 

* ACCREDITATION 
PRIORITIES 
OBJECTIVES NOT 
COMPLETED 
PERSONAL COALS 
DISTRICT GOALS 
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DEVELOPED BY: 

COMMUNITY 
SUPERINTENDENT 
SCHOOL BOARD 
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LEARNING PLAN 
COMMITTEE 
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COMPOSITION 

* PRINCIPAL 

* BASIC SKILL SPEC 

* OTHER SPECIALIST 

* GRADE LEVEL REP 

* PARENTS 
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SCHOOL 




* THE COMMITTEE 

* REVIEW GOALS 

* ANALYZE DATA 

* DETERMINE NEEDS 

* WRITE OBJECTIVES 

* THE ACTION PLAN 

* STAFF PLAN 

* GAIN APPROVAL 



+ + + + + 
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mm. 

DISTRICT GOALS 

SUPT, GOALS 
DIV. HEADGOALS 
PRINCIPAL GOALS 
LEARNING PLANS 
TCHR. LESSON PLAN 
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MODEL 

DEVELOP GOALS & 
OBJECTIVES 

DEVELOP AN EVAL- 
UATION METHOD 

PLAN STRATEGY 

IMPLEMENT - DO IT! 

EVALUATE 
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NO! 



"I WANT YOU TO 
FEEL GOOD ABOUT 
YOURSELF, BUT I 
ALSO WANT YOU TO 
LEARN." 



"WHEN YOU LEARN 
YOU FEEL BETTER 
ABOUT YOURSELF." 
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MUST HAVE A PLAN 
BASED ON GOALS! 

OFTEN THESE GOALS 
INVOLVE CHANGE 

IT IS INHUMANE TO 
LET THIS CHANGE 
OR TEACHING 
PROCESS GO TO 
CHANCE! 
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"IT'S GOT h NICE 
LOW READING 
LEVEL." 

RESULT! 

LESS VOCABULAP.LY 
LESS THINKING 
LESS WRITING SKILL 

"BUT YOU FEEL 
BETTER ABOUT 
YOURSELF." 
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WHY IS THIS 
IMPORTANT? 

5 HRS PER DAY 

X TIMES X 

12 YEARS 

OF FORMAL SCHOOL 
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THE QUALITY OF 
A SCHOOL CAN 
ONLY BE 
JUDGED 
BY THE DEGREE 
TO WHICH IT 
ACCOMPLISHES 
IT'S GOALS. 
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FWSB GRADE 9 TEST SCORES 
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RJSD GRADE 6 TEST SCORES 
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. REMEMBER THE 
GOOD OLD DAYS 
WHEN TEACHERS 
WENT HOME HAPPY 
AND THE KIDS WENT 
HOME TIRED! 
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NWASC 
ACCREDITATION 

1980-87 

3 HIGH SCHOOLS 
1 ALTERNATIVE SCHL 
5 JR HIGH SCHOOLS 
18 ELEM. SCHOOLS 

THE LARGEST 
SYSTEM IN THE 
NWASC TO BE FULLY 
ACCREDITED 
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NAUTILUS ELDCHTARY 




THE ACTION PLAN 
ACTIVITIES 
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ANALYZE TEST DATA 



ANALYZE WEAKNESS 





LONGITUDINALLY 
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The Rr le of Evaluation and 
Assessment Da ~ in Managing a Large District 

Matthew Prophet 
Portland Public Schools 



Introduction 



Modern computerized data and information systems are exciting and 
essential new tools for the effective pursuit of excellence and equity in 
tray's schools and school districts. The greater the challenges we face as 
educators, tiie more pressing is the need for data-based decision making to 
help us change for the better and bo accountable to our patrons and the public. 

For at least two score years enlightened school system administrators and 
teachers have been laboring to create, maintain, and use comprehensive data 
and information systems for the improvement of schooling at both the 
instructional and mansiement levels. The overall goal of these efforts has 
been to apply the science of data-based decision making, born in the turmoil 
of ~he Second World War, to our basic mission of supporting the magic moment 
of learning by individual students. These ^ffrots have been driven by the 
hope that by collecting, analyzing, reporting, and using accurate and 
appropriate information abcnt stude s, as well as about classroom, school, 
and discrict support systems, we could use the scarce resources entrusted to 
us to create effective, equitable and efficient self-renewing school- learning 
communities. This hop* is happily at long last beginning to be realized. 

Emerging tn ta and Information Systems in Portland 

In the Portland schools and district offices we are exploring, 
developing, and applying a wide variety of new data and information syrtems 
for district and school improvement and accountability. The framework or our 
district's efforts to use data-based systems to improve education is our 
Comprehensive Instructional Technology Plan. In April, 1983 the Board of 
Education approved and adopted the district's first such plan. Since then it 
has been annually evaluated, refined, and extended. Through this plan our 
policymakers, professionals and patrons shape our efforts to develop and 
utilize modern computerized systems to support improvement of schooling and 
school management through data and information. Coordination responsibility 
for this plan rests with our Department of Information Services. This 
department has broad leadership and support responsibilities for the 
collection, validation, and maintenance of district data and for thr 
development, implementation, and refinement of computerized information 
systems. 

The Comprehensive Instructional Technology Plan lays out the following 
1985-8J goals for improving cor district's use of data and information systems 
to support school improvement and accountability: 

• Tb continue efforts for distributing data processing and information 
services to the schools 

During the 1984-85 school year the Data Processing Department implemented 
on-line processing workstations in 13 additional schools so that more 
than half of the district^ students *ere served by on-line computer 

year an additional 25 



^ ySUr, ,,<,ptbct*iira#j>M^ year an 



schools are being provided with on-line services. These workstations put 
at the fingertips of building staff student data ranging from 
identification and demographic data to performance data such as grades, 
test scores, attendance, and courses. 

• To continue to provide training and strong staff support to schools and 
offices 



During 1984-85 seven data processing specialists and three curriculum 
specialists were available for support activities. In 1985-86 the data 
processing specialists have been organized into a user services unit to 
provide even more coordinated support. The Curriculum and Educational 
Media departments are also refining and expanding support in the areas of 
teacher training, software evaluation, and technology- curriculum 
development. This represents a significant commitment to our schools and 
offices in assisting them in the development of their skills to utilize 
computer technology to its fulleut. 

• To explore the capability of the microcomputer as a tool to a ssist the 
individual teacher in classroom management 

One elementary school has been selected for a pilot project in which a 
microcomputer and printer, along with appropriate software, will be 
provided to every teacher. The need to reduce the workload of teachers 
in the face of increasing requirements on their time and teaching load 
requires help from this technology. 

• To continue to explore and develop data communications possibilities and 
alternatives 



In 1985-86 we plan to complete much of the conversion to more efficient 
telecommunications systems for many of the schools as well as for the 
administrative and support services in the Education Service Center and 
the Child Development Center. 

The last three goals of the Comprehensive Instructional Technology Plan 

are: 

• To explore the possibility of upgrading the district's computer printing 
capabilities 

• To continue to improve the district's computer graphics and mapping 
capabilities and 

• To begin the conversion of the district's Central Processing Uni t ' o an 
even more powerful and flexible resource. 



The Comprehensive Instructional Technology Plan and its goals guide our 
electronic technology-based efforts in the areas of policy and management; 
instruction and instructional support; research, evaluation, and testing; and 
computerized educational support systems. 
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Policy and K lagement 



Within the Information Services Department is the Office of Management 
information Services which is the distribution arm of information from the 
department, and which i<* responsible for providing me and my staff with 
management information and analyses necessary for planning and monitoring 
-"istrict operations and for developing policy recommendations. That office 
also works with the Budget Office to ensure district-wide input and 
involvement in the planning/budgeting cycle. Moreover, schools and programs 
receive valuable management information tnrough routine reports and responses 
to their special requests. 



Our Office of Instructional Technology was created to spearhead our 
district's efforts to use modern technology to improve education by helping 
our students learn more effectively and efficiently. That office is piloting 
exploratory efforts in such areas as computer-controlled video disc 
applications, writing analyzers, computer art, and the use of computers in 
science and social studies. 

One mjor innovative school-b&sed information systems effort we are 
exploring is the Computer-Managed Instruction (Q4I) project. In this system, 
developed by the Curriculum and Information Services departments, when student 
reading progress assessments are desired by teachers, tests are selected from 
the curriculum guide and administered utilizing a specially designed scan 
sheet for marking the answers. Answer sheets are taken to the office where 
they are processed and reports are produced. From these reports the 
instructor has valuable information quickly in hand concerning the progress of 
a student towards his or here learning objectives. Information from these 
reports is useful in determining whether reteaching is required, regrouping 
students is necessary, or whether to continue the instructional process as 
planned. 



Another department that plays a central role in fostering data-based 
decision making at the district, program, classroom, and individual student 
levels Is our Department of Research and Evaluation. The chief mission of 
this offioe is to add to the quantity, objectivity, validity, reliability, and 
accessiblity of the data we use to make decisions to help students advance and 
programs improve. They do this primarily by providing testing, evaluation, 
and research reports and technical support to help assess and improve student 
instruction and achievement, as well as to evaluate and enhance educational 
support and management services. These reports and services are provided to 
teachers, students, parents, support staff, building and program managers, 
central administrators, policymakers, and citizens. They are designed to 
facilitate targeting instruction on student needs and to help improve 
programs, policies, and resource allocations. 



Instruction and Instructional Support 



Research, Evaluation, and Testing 
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In addition to operating successful programs ot testing and evaluation 
our Research and Evaluation Department leads an ongoing program of 
collaborative research and development aimed at continuously improving our 
capacity to meet student achievement data and information needs. Some of the 
new helps to decision naking based on data about student performance that are 
debuting this year are: 

• New mathematics and language arts achievement testing systems completely 
revised to reflect improvements in the curriculum, including the greater 
emphasis on higher order skills such as problem solving. 

• Development and piloting of new systemwide testing systems in the 
critically important areas of science and direct writing. 

• Streamlining and graphically enhancing the test results reporting system 
in order to provide better support for appropriate interpretation and use 
of research, evaluation, and testing results. 

• Offering districtwide a school-based microprocessor test reporting 
system. Over the past two years, a group of seven principals, along with 
the Data Processing Department and the Research and Evaluation 
Department, have worked to develop this computer system for local 
building controlled reporting and analysis of test data. The pilot 
system began with four goals in mind. We wetc interested in finding a 
program that would run on building microcomputers that would accomplish 
the following: 

1. Provide a complete individual student test history to building staff 
immediately on request. 

2. Produce test reports by instructional group. 

3. Provide analyses of longitudinal student group data when and as the 
building needed them, and 

4. Improve the turnaround time of test reports. 

We now have a program which gives local buildings the ability to meet 
these four goals and we have offered it to all schools in the district. 

• Initiating a pilot of a school-based computer adaptive testing system 
which allows building personnel to continuously monitor the progress of 
students as they advance through the basic skills curriculum. This 
system involves putting a sufficient bank of field-tested items inside a 
computer along with the requisite software to build a unique, 
individualized test for each student at the time when building staff feel 
it is needed. 

The advantages of this system include: 

• Increased measurement accuracy 

• Increased testing flexibility 
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• Improved use of testing as an integral part of the instructional 
process 

• Enhanced test security 

• Decreased testing time 

• Increased ability to measure high-level educational goals such as 
problem solving 

• Immediate feedback of results 

Our Research and Evaluation Department has mounted a pilot CAT program 
this fall in cooperation with the Information Services Department, directors 
of instruction, and principals, the purpose of this pilot is to gain the 
information necessary to design a coct-effective CAT system that will serve 
the future testing needs of all our students and our schools. 

Computerized Educational Support System 

Our Data Processing Systems Development staff is continuously developing, 
maintaining, and enhancing a wide variety of computerized educational support 
systems including: Payroll, Personnel, Maintenance , Transportation, 
Educational Media, Financial Planning and Accounting, Facilities, Food 
Services, and Student Information. 

These systems utilize the latest technology such as on-line transaction 
processing, data base management, optical scanning, interactive terminals, and 
networked microcomputers. These sophisticated data and information systems 
are all ultimately dedicated to helping us assist students to learn. 

Benefits of Such D*ta and Information Systems 

The uses to which such emergent innovative data and information systems 
can be and are being put in Portland and elsewhere about this state and this 
nation are great and greatly exciting. They include: 

• Targeting instruction on the needs, abilities, readiness, and 
characteristics of individual learners. 

• Grouping and placing students so that their learning needs may be met 
most effectively and efficiently. 

• Better and easier classroom, building, and district management resulting 
in better decisions at lower cost, freeing time, and other resources to 
be spent on instruction and direct learning support and leadership. 

• Prompt, accurate, and thorough evaluations of program cost effectiveness. 

• Timely and thorough data to support policy formulation a d monitoring. 
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Tested theories of effective education now reveal what we must do to 
imporve schools and to help every child learn as much as he or she can as 
effectively and efficiently as possible. They indicate that our educational 
leadership must support the development and learning environments in which the 
following sorts of things happen for each of our students: 

• His or her current, most pressing learning needs within a well planned 
curriculum must be identified. 

• The student must be helped to set clear, relevant, attainable learning 
objectives to et those needs. 

• He or she mast be expected to succeed in attaining the learning 
objectives and must want to learn them. 

• The student must receive individualized instruction directly related to 
the learning objectives designed to meet his or her current learning 
needs. 

• The learner must use the time allocated for instruction to work intently 
and seriously on the task of learning. 

• The student must know when he or she has succeeded and when not, and must 
experience a reinforcing sense of accomplishment and achievement as a 
result of knowledge of success. 

• The learner must receive and return a sense of caring, personal concern, 
interest, respect, and commitment which provides the psychological 
support necessary to want to learn and to work to learn, and finally 

• The student must receive and accept parental and community support and 
encouragement for success in learning. 

The main barrier to our putting such models of effective instruction and 
education into practice up until now has been the lack of accurate data and 
information about: 

• Each student's individual learning needs. 

• What learning activities and experiences ate matched to diagnosed student 
needs and to established learning objectives and how to help the student 
engage in s»,ch tailored instruction in a timely fashion. 

• When the student has mastered the objectives and is ready to move on. 

• The degree of overall success of staff and programs in promoting student 
learning, and 

• What is and is not working to help students learn. 
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We are now, however, at long last beginning to evolve the comprehensive 
electronic data and information systems needed in order to create the more 
effective, equitable, and efficient education systems required for real and 
meaningful educational reform and even reinvention of schooling. 

Conclusion 

John Holt has chided the current educational reform movement for taking a 
horese and buggy system, repainting the buggy, paying the driver more, keeping 
the passengers in it longer, and foolishly expecting it to go faster and 
better. 

By placing modern data and information systems technology in the service 
of school improvement and accountability, however » wc are unhitching the good 
but tired old horse, putting a powerful engine in the educational buggy, 
turning the driver loose, and riding off swiftly toward meeting our goal of 
helping every child become everything he or she can and wants to be. I, for 
one, am immensely pleased and excited to be along for the ride on this 
historic leg of the educational journey. 
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